Felder, Richard, "What Do They Know Anyway? 2. Making Evaluations Effective."
Chem. Engr. Education, 27(1), 28-29 (Winter 1993).
Richard M. Felder
Department of Chemical Engineering
North Carolina State University
Raleigh, NC 27695-7905
Two columns ago I tried to persuade you that contrary to conventional faculty lounge wisdom, student evaluations provide reliable indicators of teaching quality: they correlate well with retrospective evaluations submitted by graduating seniors and alumni and tend to be higher for instructors whose students do best on common examinations. The question is not whether the evaluations mean anything---they clearly mean a lot---but how they should be structured to do the most good.
Following are some ideas for constructing, administering, and interpreting evaluations, starting with the simplest forms and proceeding to methods that take more work to implement but are more likely to improve teaching quality. For more suggestions and summaries of research on teaching evaluation, see Reference 2.
Collect overall course-end ratings of instruction. For example, "Rate the instruction you received in this course on a scale from 1 to 5, with 5 being the highest response."
Ratings of this sort are most effective when the numbers on the response scale are clearly defined. Definitions like "excellent," "above average," "fair," etc., don't do it: these terms are ambiguous and when they are used a very broad performance range tends to be lumped into "above average." You can get greater discrimination with a variation of the following instruction:
When responding, use as a basis of comparison all of your previous high school and college teachers. A response of 5 denotes one of the three or four best you've ever had; 4 = top 25%; 3 = 40-75%; 2 = bottom 40%; and 1 = one of the three or four worst you have ever had.
An instructor whose average rating is close to 5 on this scale is clearly doing a superb job and deserves nomination for an outstanding teacher award, and serious problems obviously exist if an instructor's rating is consistently close to 1. Ratings close to 4 indicate commendable teaching performance and ratings close to 2 suggest the need for corrective measures.
Collect ratings of individual aspects of instruction.
To get the most out of a course-end evaluation, supplement the overall rating with ratings of specific aspects of teaching performance, such as clearly stating expectations, providing frequent examples, repeating difficult ideas, pointing out practical applications, answering questions thoroughly, preparing tests that reflect course content and emphasis, etc. (General questions about the instructor's preparedness and knowledge of the subject tend to be less useful.) The responses help identify areas of weakness and may provide ideas about how to improve teaching in the next course.
To be sure that the evaluations reflect a true cross-section of student opinion, administer and collect them in a single class session rather than counting on students to return them later. Results of evaluations for which the return rate is less than a minimal percentage should be regarded with deep suspicion: the recommended minimum is 50% (classes of 100 or more), 66% (50-100), 75% (20-50), and 80% (<20).[2, p.89]
Collect evaluations midway through a course rather than waiting until the end.
If the goal is to correct teaching problems and not just to identify them, find out what the problems are while enough time remains to do something about them. Ask open-ended questions on midcourse evaluations, leaving plenty of space for the responses:
Collect evaluations from small groups rather than from every student.
One problem with individual evaluations is that many of the responses may reflect isolated gripes rather than widely-held opinions. Another is that students may be fearful of offering negative criticism while a course is still in progress, even if the evaluations are anonymous (as they should be). A good way to counter both of these problems is to collect evaluations from groups of four or five students rather than from individuals. The students in a group should spend 5-10 minutes discussing the three questions given above, then prepare a collective evaluation that only includes points agreed upon by several group members.
Interview student representatives.
Designate certain students as representatives of subgroups within the class. At one or more times during the semester, meet (or ask a colleague to meet) with the representatives to share the concerns of their constituents and to discuss possible measures to correct perceived problems. This procedure tends to generate constructive criticism at a level rarely attained through written evaluations and also gives students a greater sense that their opinions are valued.
Use a variety of sources of feedback.
Collect retrospective teaching evaluations from graduating seniors and alumni. Have faculty colleagues observe your teaching and provide feedback. Have one of your classes videotaped and review the tape (brace yourself---you may not be thrilled by everything you see).
Work with an instructional consultant to interpret student feedback and plan teaching improvement strategies.
It is one thing to know that some students consider you a poor lecturer or think your tests are unfair and quite another to know what to do about it. Many universities have instructional consultants whose job is to help faculty members improve their teaching. These people can provide a variety of services, such as helping design and administer evaluation questionnaires, interviewing classes or groups of students about their perceptions of the instruction, observing and critiquing live or videotaped lectures, and working with instructors to help interpret evaluations and plan corrective strategies. If no one like this is available on your campus, ask a faculty colleague with a reputation as an outstanding teacher to work with you.
Can properly interpreted student feedback improve teaching? The research suggests that it does. In one study, instructors who received no feedback in the first half of a course received average end-of-term ratings in the 50th percentile of the population studied; instructors who received feedback scored in the 58th percentile; and instructors who got both feedback and instructional consulation scored in the 74th percentile. While midcourse evaluations are not guaranteed to improve course-end ratings and the teaching they reflect by that much, they are bound to have positive effects. A university, school, or department seeking to raise the level of its teaching program (e.g. as part of a TQM initiative) might well consider instituting midcourse evaluations and providing instructional consultation as a strong first step.