Draft Qualitative Criteria for the Evaluation of Graduate Programs November 17, 1995 1 General Considerations Because graduate programs at Northeastern University have legitimately diverse missions, no single criterion can be used to evaluate them. How- ever, it is possible to set forth general principles that should govern such an evaluation. o The evaluation criteria must be sufficiently broad to encompass the variety of contributions that different programs may make to the Uni- versity's varied missions. o The evaluation criteria must be sufficiently broad so as to be adaptable to different units in program-specific, discipline-specific ways. o Any evaluation must begin with self-study by the programs, depart- ments, or offering units. Only these units have the detailed knowledge necessary to understand their status, roles, and contributions to the University. o Any evaluation process subsequent to the self-study must grant the unit an opportunity to review the conclusions of that process and to correct errors of fact or perception that may have arisen. o Any evaluation process that envisages serious reallocation of resources among programs should include three sets of criteria: criteria designed to evaluate the current state of a program, additional criteria that must be satisfied to justify program expansion or enhancement, and additional criteria that must be met to justify program reduction or elimination. o In keeping with the principles of the Strategic Plan, decisions about reallocation of resources should be made at the lowest possible level whenever possible. o Any University-wide evaluation process must be adequately funded at the unit, college, and University levels so that the evaluation process itself will not become a serious drain on personnel resources at each level. In this document, we list a number of criteria for evaluating and allocating resources to graduate programs. These criteria are not ranked. They fall, however, into two basic groups: those that address graduate program quality and overall success in relation to the program's own internal goals and appropriate external disciplinary standards, and those that assess the program's contributions to and fit within the strategic missions of the the College and Northeastern University. For each of these criteria, we list questions in three groups: those that assess the current state of the program, those that must be answered to justify program expansion or enhancement, and those that must be answered to justify program reduction or elimination. Not every question is applicable or relevant to every program. In every case, these criteria must be applied only insofar as they are relevant to the program being evaluated. 2 Program Quality These criteria address graduate program quality and overall success in re- lation to the program's own internal goals and appropriate external dis- ciplinary standards. In most cases, these questions require answers that evaluate the program both in an absolute sense and in comparison to other programs in the discipline. 2.1 Students For evaluation of the current state of the program: Does the pro- gram meet the needs of some identifiable class of students? Is the number of students enrolled consistent with the minimum needed to maintain a crit- ical mass and with the maximum that the faculty can supervise? Do the students enrolled in the program meet the program's goals for quality? Do the students enrolled in the program meet the program's goals for diversity? How do the quantity and quality of students in the program compare with that of other programs in the discipline? Have students been successfully placed in the kind of positions initially projected for graduates of the program? Have they passed applicable licens- ing or certification exams? For expansion or enhancement of the program: How would pro- posed program expansions or enhancements meet student needs? How would they raise student quality or improve the diversity of the student body? What additional resources (faculty, staff, equipment, space) would be nec- essary to serve an expanded or changed student body? For reduction or elimination of the program: Have student qual- ifications consistently failed to meet projections over time? Has the pool of graduate students sunk below critical mass in terms of either quantity or quality? Have graduates consistently failed to meet the program's placement goals over time? Would scaling back the program allow for more selective admissions? 2.2 Curriculum For evaluation of the current state of the program: Is the curriculum current with professional standards? Is it innovative in ways likely to attract good students over our competitors? How does the program fit in with related programs in the department or College? Has the program positioned itself to take advantage of consortial or other resource-sharing opportunities in the area? Does the program offer sufficient high-quality internships or practica (for clinical/professional curricula) or research opportunities (for research-oriented curricula)? For expansion or enhancement of the program: How will expan- sion or enhancement improve the curriculum? For reduction or elimination of the program: Has the program failed to cure previously identified curricular weaknesses that were within its power to correct? Would a reduced program help cure such weaknesses and be academically sound? Are its weaknesses so systemic as to be insoluble by expansion, enhancement, or reduction? 2.3 Faculty For evaluation of the current state of the program: Are faculty qualifications appropriate to meet the stated goals of the program? Is the fit between demonstrated faculty interests and the goals of the program a good one? Have faculty interests and qualifications kept pace with the realities and goals of the program over time? Does the faculty of the program meet the program's goals for diversity? For expansion or enhancement of the program: If new resources are proposed, how will they better the fit between faculty strengths and program goals? For reduction or elimination of the program: Do insufficient fac- ulty positions exist to operate the program? Are faculty unable to sustain the momentum of the program or project new directions for program en- hancement and development? Can existing faculty be utilized better in some other program? 2.4 Staff and Infrastructure For evaluation of the current state of the program: Are support staff, space, and equipment adequate to meet the unit's needs? For expansion or enhancement of the program: What shortcom- ings will be alleviated by additional staff, space, or equipment? If the pro- gram is to be expanded in general, what additional staff, space, or equipment will be needed to make the desired impact? For reduction or elimination of the program: If the program is to be reduced, can existing staff, space, and equipment be better utilized in other ways? 2.5 Program Reputation These questions address the program's prestige and reputation outside the University. For evaluation of the current state of the program: What is program's contributions to prestige outside the University as reflected in the following areas: 1. Placement of graduates, either as practitioners or as teacher/scholars. 2. Dissemination of knowledge, including scholarly publication, advance- ment of clinical practice, technology transfer, or performance/exhibition. 3. Success in obtaining external funding for research, training, commu- nity programs, etc. 4. Service to the community. 5. Other external recognition and reputation, including surveys of schol- ars, community leaders, or corporate leaders. Whenever possible, such evaluation should an evaluation of the program in comparison to discipline-specific norms. What are the dynamics of the program? Is it improving, holding steady, or declining? For expansion or enhancement of the program: How will the pro- gram's contributions in these areas be materially increased by the infusion of new resources? Will the proposed enhancements or expansion eliminate barriers block- ing the program from excellence? Has the unit made good use of avail- able resources, including inter-institutional opportunities? If new curricular or program resources are proposed, how will they enhance the program's innovative reputation or general competitive profile? Does the proposed expansion or enhancement leverage the strengths of existing programs? For reduction or elimination of the program: Will the loss of contributions to prestige in these areas be minimal with program reduction or elimination? Will eliminating a program materially hurt an identifiable group of alumni? Will reduction or elimination free substantial resources for more pressing University priorities? Are external ratings based on accurate information? 3 Relation to College and University Missions 3.1 Contributions to undergraduate education For evaluation of the current state of the program: How does the pro- gram contribute to undergraduate education? Does the program offer career models for undergraduates, and/or encourage undergraduates to consider graduate education? Does the program enhance undergraduate experiences of diversity or multiculturalism? Where applicable, do doctoral students play similar roles for masters' students? For expansion or enhancement of the program: How would pro- gram expansion enhance its contributions to undergraduate education in the arenas listed above? For reduction or elimination of the program: Are programs com- pletely estranged from undergraduate goals or do they actively work against undergraduate educational needs? 3.2 Financial contributions For evaluation of the current state of the program: Estimate the pro- gram's net financial contribution to the University. Such an estimate should include at least the following considerations: 1) direct tuition revenue; 2) return from external grants; 3) direct expenses; and 4) saved resources from utilizing SGAs for necessary tasks rather than more expensive personnel. Are projected contributions in line with disciplinary norms and with the strategic role of the program within the College and University? How do these contributions compare with those in the original program proposal or more recent projections? For expansion or enhancement of the program: How will expan- sion or enhancement yield significantly greater returns? For reduction or elimination of the program: Has the program's financial performance over time consistently failed to meet projections and disciplinary norms? Could the program be scaled back in such a way as to improve its rate of return on investment? 3.3 Contributions to strategic missions For evaluation of the current state of the program: How does the program fit within the strategic mission of the unit, College, and University? How does the proposed program significantly enhance the College's and the University's ability to carry out its strategic missions? What does it contribute that other programs do not or cannot? For expansion or enhancement of the program: How does the pro- posed expansion or enhancement significantly increase the program's con- tributions to the strategic missions of the unit, College, and University? For reduction or elimination of the program: Will the loss to the College's or the University's ability to fulfill its strategic mission be minimal if the program is scaled back or eliminated?