David Schleich, PhD
In recent months we have been considering the value to our naturopathic colleges to develop an institutional research (IR) department. It is important in this process to take into account what Middaugh describes as “possible interactions of measures” in a well-known model called the “input-process-output,” or IPO model. These are as follows:
- Inputs (college selection survey; advanced placement tests; analysis of student characteristics; financial aid)
- Intersecting inputs and processes (survey of housing needs; surveys of special population needs; survey of provincial and national demographics impacting on entry to practice)
- Process (student satisfaction surveys, faculty and staff satisfaction surveys, student attitude surveys, faculty and staff attitude surveys, program evaluation)
- Intersecting inputs, process and outputs (longitudinal studies of overall effect of college regionally, nationally, internationally; retention analysis; career plans and employment; portfolio assessment)
- Outputs (post-graduate credential awarded; NPLEX and entry to practice/licensing results; alumnae surveys)
Because a critical element of any assessment activity within this framework is “high quality and, therefore, useful” information (Middaugh, 1994, p. 46), a general set of steps is important to incorporate into the design of the IR function. Davis (1989) recommends the following general approach for any assessment.
- Focus the evaluation by identifying goals and restraints from the client
- Identify stakeholders and audiences
- Generate questions of interest to the stakeholders
- Refine and limit questions through negotiation with the vested parties
- Determine the methodology: For each question specify the instrument or data source, the sample from whom data have been or need to be collected, the time frame for collection of data, methods of analysis and the intended use of the results
- Communicate the findings to stakeholders in ways that they can use the results
A naturopathic college IR department would not only make use of the guidelines and the IPO model outlined above, it could be a centralized model at our colleges. Slark (1992) describes a successful model of this kind established in the 1990s at Rancho Santiago College. He describes its IR program as “an effective, integral part of [his] college’s decision-making process” (p. 5). Slark goes on to explain that theirs is a model that “assumes a broad base of responsibility for institutional research” (p. 6). The model includes formal relationships between the IR office and other departments. As such, the “research activities become credible because they reflect the institutional priorities established by his college’s councils, the chancellor and the cabinet. The director of research and planning is an institutional leader and active member of the college team, rather than an isolated actor” (pp. 6-7). Strong co-leadership of projects is implicit in the structure. This centralized approach “has the obvious advantage of closely coordinating an institution’s research activities” (p. 10). Further, it avoids duplication, reveals gaps in the institution’s research knowledge and “allows a focus on the changes that will result from the research” (p. 11). The disadvantage of the research function being separate and its director perceived sometimes as an outsider can be overcome by strong connections to the programs and services being assessed.
Principal Functions
The following principal functions can be the basis for a naturopathic college IR department: research/evaluation services; planning and market assessment services; management information services; technical support and consulting services.
Research/Evaluation Services
- Systematic, biannual program review
- Annual student evaluation of instructors and college services, including the library, cafeteria, student services, residences/dormitory, plant and property and classroom/clinical experience
- Annual student evaluation of counselors and advisors
- Ongoing patient evaluation of clinical services, information services, customer service
- Conducting of needs assessments and feasibility studies
- Facilitation of educational research support services in conjunction with the office of the associate dean of research and graduate programs
Planning and Market Assessment Services
- Periodic surveys of college students and graduates
- Periodic surveys of the profession locally, provincially, nationally
- Continuous monitoring of student and patient demographics
Management Information Services
- Monitoring and reporting of college enrollments
- Coordination and completion of provincial, federal and other reports and surveys
- Provision of data for college and committee initiatives and projects
- Assessment of institutional effectiveness and student outcomes
Technical Support and Consulting Services
- Technical support/consulting for faculty research
Within these parameters, the following core indicators will be routinely assessed and reported on:
Student Services
- Distribution of entering and exiting grade average
- Acceptance and yield rates: entry from undergraduate background; professional second entry; entry from graduate background; special admission
- Percent of U.S., Canadian, international students
- Percent of students holding scholarly awards
- Student retention, attrition and progress toward the ND credential
- Assessment of student services provided: recruitment, financial aid, counseling, student life
- Time to graduation
Academic Services
- Full-time faculty: graduate and professional credentials
- Library resources: volumes acquired/held; online services; utilization rates; total spending
- Percent of faculty holding scholarly awards
- Percent of faculty publishing
- Percent of faculty with education credentials
- Courses listed/courses actually scheduled
- Percent of faculty with doctorates
- Percent of faculty with ND doctorate (first professional degree)
- Percent of faculty with other healthcare profession designation
- Percent of faculty licensed/registered to practice as NDs
- Instructional load: teacher/student ratio; student contact hours; class size; ratio of sections to courses
- Academic staff on research or clinical practice leave
- Research grants per professor
- Research grants and contracts as a percentage of operating revenue
- Educational technology: audiovisual collection per student; audiovisual collection per course; audiovisual collection with copyright; multimedia classrooms; courses with computer applications
Plant and Property
- Space allocation/purpose
- Space utilization: classrooms, teaching laboratories, research laboratories, academic department offices, central administration offices; library, college support and services; maintenance shops
Finances
- Comparison of growth rate in E&G revenues compared with E&G expenditures
- Ratio analysis: revenue contribution ratios; expenditure demand ratios
- Salary and compensation comparisons
- Annual reports on the economic status of the profession
- Salary equity analyses
- Semester-based and annualized instructional productivity analysis
- Instructional productivity
- Research and service productivity
- Cost analysis: expenditures by object and function: direct instructional cost per student credit hour (by semester and annually)
- Administration productivity
IR Implementation Guidelines
Middaugh et al. (1994) propose basic guidelines for the implementation of an institutional research capacity, all of which are relevant and useful for us:
- Firmly establish the centrality of our naturopathic colleges’ respective offices of IR “in coordinating campus databases and disseminating institutional data”
- Establish a friendly, professional relationship with the manager of management information systems and his or her staff
- Wherever possible, obtain data from primary sources
- Minimize the use of surveys where possible and, where utilized, favor commercially prepared, professionally prepared instruments
- Write reports that are easy to understand, brief and concise, and that are focused on influencing change, always moving from information to improvement
- IR should never be complacent; the IR department must constantly seek new ways to generate factual information that leads to concrete policy activity
With the above design elements, framework and implementation guidelines in mind, let us now consider how to activate an IR capacity at our colleges. See the accompanying article, Suggested Model IR Implementation Project.
Purpose
To generate a framework for implementing an ongoing, comprehensive assessment of our colleges’ operational and strategic activities by utilizing the resources of systematic IR.
Theoretical/Conceptual Framework and Rationale
Basing the development and implementation of the project on the theoretical considerations of Volkwein and Peterson (1999) in particular about IR, our colleges could strike steering committees to design and introduce into the culture an office of IR that will have a clear mandate and be properly resourced. We can turn to the work of Middaugh et al. (1994) for pragmatic resources to assist us in the design and implementation process. Of particular use can be the work of Peterson et al. (1991), Saupe (1990) and Norris and Poulton (1987) on the theory and application of IR; Society for College and University Planning (SCUP) for a focus on planning issues; Dunn (1989) for assistance in those assessment elements relating to electronic media and information sharing; Krol (1992) for help with Internet considerations; Kalton (1984) and Suskie (1992) for “profitable references” about survey research; Scannel (1992) for pragmatic analysis of admissions and financial aid processes; Teeter and Brinkman (1992) for guidelines about accessing and utilizing peer institutions for IR; Astin (1993) and Tinto (1987) for references on student development; Halpern (1987) and Noel et al. (1985) on student retention; and finally, Gerlinda Melchiori (1989) on alumni survey research.
The steering committee could incorporate existing data collection files and efforts into its resources and will begin immediately to design new procedures as necessary. Finally, the steering committee could establish at least two pilot projects during the first year at any college choosing to implement this approach. For example, the first project could be in the student services area, and the second in the clinic operations area.
Criteria for Assessment
There are four key indicators of the successful design and implementation of this IR strategy:
- Staff, faculty, student consensus on the model developed
- Successful design by the end of month one of an eight-month project
- Successful implementation of a multi-phase rollout strategy by the end of the ninth month of the project
- Articulation of operationally defined outcomes: appropriate use of measurable behavioral terms
Significance and Implications
If our colleges are to take their proper place among the distinguished colleges and universities of naturopathic medicine evolving in North America, they must develop capacity for IR that can assist in continuous improvement objectives, can help respond to external pressures from primary healthcare professionals and their agencies and associations and from government, and that can help celebrate successes by reporting outcomes accurately and reliably as they are achieved.
Anticipated Outcomes or Expectations
The naturopathic community will welcome an IR resource, and especially value objective, reliable data describing the efficacy of its inputs, processes and outputs. In particular, the inconsistency of the profession’s response to the state and provincial variations in health policy and the reforms, which are part of the healthcare landscape, can be more effectively countered with reliable data about the quality of professional education that nourishes the next wave of new professionals at the point of their entry to practice.
David Schleich, PhD is president and CEO of NCNM, former president of Truestar Health, and former CEO and president of CCNM, where he served from 1996 to 2003. Other previous posts have included appointments as vice president academic of Niagara College, and administrative and teaching positions at St. Lawrence College, Swinburne University (Australia) and the University of Alberta. His academic credentials have been earned from the University of Western Ontario (BA), the University of Alberta (MA), Queen’s University (BEd) and the University of Toronto (PhD).
References
Astin A: What Matters in College. San Francisco, 1993, Jossey-Bass.
Davis BG: Demystifying assessment: learning from the field of evaluation. Achieving Assessment Goals Using Evaluation Techniques, In PJ Gray (ed), New Directions for Higher Education, No. 67. San Francisco, 1989, Jossey-Bass.
Dunn JA: Electronic media and information sharing. In P Ewell (ed), Enhancing Information Use in Decision Making New Directions for Institutional Research, No. 64. San Francisco, 1989, Jossey-Bass, Publishers.
Gray ST: Evaluation With Power: A New Approach to Organizational Effectiveness, Empowerment, and Excellence. San Francisco, 1998, Jossey-Bass.
Halpern DF (ed): Student outcomes assessment: what institutions stand to gain. In New Directions for Higher Education, No. 59. San Francisco, 1987, Jossey-Bass.
Kalton G: Introduction to Survey Sampling. Beverly Hills, 1984, Sage Publications.
Krol E: The Whole Internet: User’s Guide and Catalog. Sebastopol, 1992, O’Reilly and Associates.
Melchiori G: Alumni research: methods and applications. In New Directions for Institutional Research. San Francisco, 1989, Jossey-Bass.
Middaugh MF et al: Strategies for the practice of institutional research: concepts, resources and applications, No. 9. In Resources in Institutional Research. The Association for Institutional Research. Tallahassee, 1994, Florida State University.
Noel L et al. (eds): Increasing Student Retention. San Francisco, 1985, Jossey-Bass.
Norris, DM and Poulton NL: A Guide for New Planners. Ann Arbor, 1987, Society for College and University Planning.
Peters R: Accountability and the end(s) of higher education, Change Nov-Dec:16-23, 1994.
Peterson MW et al: Theory and applications of institutional research. In WR Fendley and LT Seeloff (eds), Reference Sources: An Annotated Bibliography. Tallahassee, 1991, Association for Institutional Research.
Peterson MW: The role of institutional research: from improvement to redesign. In JF Volkwein (ed), What Is Institutional Research All About? A Critical and Comprehensive Assessment of the Profession. New Directions for Institutional Research, No. 104. San Francisco, 1999, Jossey-Bass.
Popham WJ (ed): Evaluation in Education: Current Applications. Berkley, 1974, McCutchan.
Saupe JL: The Functions of Institutional Research (2nd ed). Tallahassee, 1990, Association for Institutional Research.
Scannell J: The Effect of Financial Aid Policies on Admission and Enrollment. New York, 1992, College Entrance Examination Board.
Society for College and University Planning (SCUP): www.scup.org, Ann Arbor, University of Michigan.
Slark J: The traditional centralized model of institutional research, New Directions for Community Colleges, No. 72:5-11, 1992.
Suskie LA: Questionnaire Survey Research: What Works. Tallahassee, 1992, Association for Institutional Research.
Teeter D and Brinkman P: Peer institutions. In M Whitely et al. (eds), The Primer for Institutional Research. Tallahassee, 1992, The Association for Institutional Research.
Tinto V: Leaving College: Rethinking the Causes and Cures of Student Attrition. Chicago, 1987, University of Chicago Press.
Terenzini PT: On the nature of institutional research and the knowledge and skills it requires. In JF Volkwein (ed), What Is Institutional Research All About? A Critical and Comprehensive Assessment of the Profession. New Directions for Institutional Research, No. 104. San Francisco, 1993, Jossey-Bass.
