By Ian Shapira and Mark Schulte

This article was written as part of an online news production course at American University in Spring 2011. It was originally published in the Observer, the AU graduate student news site.

When Alicia Rodrigues fills out professor evaluations at the end of her graduate courses at American University, she often feels like the questions do not probe deeply enough about the teachers’ talents. Rodrigues, who also works as the university’s student media coordinator, said the evaluation questions are too generic and don’t prompt students to rate professors on specific skills.

“I feel like there should be more pointed questions like, ‘Did the professor teach you to write an effective press release?’ Or, ‘Were you able to learn the components of communicating with a reporter?’” said Rodrigues, who is wrapping up her weekend graduate degree in public communications. “If you’re truly going to be evaluating a course, then the questions should be tailored around those kinds of questions.”

An American Observer analysis of the School of Communication weekend program’s professor evaluations revealed other potential concerns including:

• An interactive data visualization, developed by the authors, shows data from the three weekend School of Communications masters programs in both static and animated views. Click the image to view.
• Students can find it difficult to locate evaluation scores on the university’s website. Distinctions can be unclear between full-time courses and the SOC’s three weekend programs: Public Communication; Producing Film and Video; and Interactive Journalism. (Students must choose either the full-time or weekend programs.)
• Prospective SOC students do not have access to the internal AU Portal where evaluations are housed to see how current students rated professors; and to comparison shop between the weekend programs, especially Interactive Journalism and Producing Film and Video.
  Some evaluation scores for some professors are missing, compromising the assessments’ overall value.
• The written evaluations — in which students can write anonymously about the course and professor — are not sent to SOC administrators, but instead are kept by the professors. Sometimes, SOC students write essays under the false assumption that administrators automatically read all of them.
• A small number of aggrieved students can give professors very low scores as payback for class difficulties, easily skewing the professors’ overall scores, which may be a factor in whether some can continue teaching.

‘Tyranny’ of evaluations
SOC administrators say the evaluations help them determine which professors excel, struggle teaching the course content, or are unpopular with students. The evaluations also spotlight instructors’ specific teaching deficiencies and can help with determining what steps professors can take to improve.

Students rate the professors on scale of 1 to 7 (worst to best) on questions ranging from whether their writing improved to whether class time was used productively.

Rose Ann Robertson, the SOC’s associate dean of academic affairs, acknowledged that student evaluations of professors can be inherently flawed.

“I think you have to take the evaluation data with a huge pinch of salt,” said Robertson, who helps oversee the school’s evaluations, along with faculty and curriculum.

Darrell Hayes, director of the Public Communication weekend program, said he believes that the evaluations are valuable, but that they can also impose a kind of “tyranny” over the school’s faculty.

“The difficulty is that these measures are almost like a life-and-death system in the university,” Hayes said. “If you ever have a few people in the class who don’t like what you’ve done, then in essence you may never teach at the university again, especially if you’re an adjunct.”

Evaluations at other colleges
The university’s evaluation system is very similar to others nationwide, said Robert Kreiser, associate secretary and senior program officer at the American Association of University Professors (AAUP). He cautioned that colleges should be careful in relying too much on the data in determining whether faculty members should be reappointed, granted tenure or promoted.

“There should be an agreement that the results would not be used as the sole or primary measure of a professor’s teaching performance or to an extent greater than the faculty believes is appropriate,” he said. “I reject the idea that students should be viewed as customers who have to be satisfied.”

Nationwide, colleges have been debating since at least the 1970s the credibility and reliability of student evaluations of professors, which he said are vulnerable to student prejudices.

“There are gender biases, beauty biases and other biases that affect scores, one way or another,” Kreiser said. “In light of these concerns, the AAUP has called upon institutions to include other means of assessing teaching performance — in class peer evaluation and reviews of syllabi, in addition to student evaluations.”

Other approaches
Robertson said teachers are allowed to choose more specific questions for the evaluations, but she doesn’t know how many of them actually do it. “It is something that I will remind faculty of in the future,” Robertson wrote in an email.
She added that some faculty members distribute midterm evaluations in which students can provide anonymous comments about how the class is going, so they can adapt. \

“I have suggested this to some faculty whose previous evaluations have posed some concern,” she said. “I don’t have statistics, but anecdotal evidence show that many faculty who do this midterm evaluation are able to address student concerns and increase their [evaluation scores].”

Robertson said a side-by-side comparison of the weekend programs would not be useful for prospective students because the classes are so different. “I think you’re comparing apples and oranges,” she said. “But I am not sure that’s how you decide what your [master’s] degree is.”

She did acknowledge that a comparison might be useful if a student was deciding between two potentially similar programs within a department, say, the Interactive Journalism and Film and Video programs.

Missing evaluation data
In some cases, professor evaluation data is missing, which distorts a professor’s ultimate score, according to the American Observer analysis.

Three professors — one in each of the SOC weekend masters programs — who taught in the summer 2010 scored a zero out of a possible seven points on three key evaluation measures. The evaluation form would suggest that out of 21 questions, nobody in any of the cohorts chose to answer in those three categories: deepening a student’s interest in the subject matter; whether or not the professor suggested specific ways to improve; and whether the course challenged the student.

For instance, in the Interactive Journalism program, backpack video professor Rob Roberts scored zero out of 7 on those three questions. On the question of whether the instructor improved students’ writing abilities, Roberts somehow netted only one response — a 7 out of 7 — while everyone else apparently did not respond at all. (Writing assignments were few in the backpack video course.) Meanwhile, Roberts’ course was among the highest-rated in the entire SOC weekend program, with an overall score of 6.5.

Robertson, the SOC associate dean, said she was not aware of any recent problems with missing data in the weekend programs. “It’s not something I come across frequently,” she said. “Sometimes, the evaluations get lost.”

High course scores, such as those for the backpack video class, reflect an SOC trend, Robertson said: the school’s courses regularly earn scores above the university mean. For instance, in the fall of 2010, the SOC’s graduate courses had a mean score of 5.73, compared the university-wide graduate course mean of 5.63, Robertson said.

Private written evaluations
For many students, though, the written evaluations are the most important part of the assessment because they allow for the most detailed analysis. But Meg Cangany, who is graduating this spring from the weekend Public Communication masters program, was surprised to learn that written evaluations were not sent to the administrators.

“That blows my mind — this entire time I thought I was writing these evaluations to the administrators and that would affect whether the professor came back and taught another course,” said Cangany, a media relations officer for Plan International, a British-headquartered non-profit that assists children in developing countries. “That’s the most important piece of the evaluation.”

The Faculty Senate voted to keep the critiques seen only by the professor because that level of privacy would free up students to be more honest, Robertson said.

Some SOC administrators said if they see that a professor’s scores on the questionnaire are low, they may ask the instructor if they can see the narrative evaluations.

“Nine times out of ten, the teachers say ‘Yes,’ when asked,” said Robertson.

Randall Blair, the director for the Producing Film and Video masters degree program, said when one of his adjunct professors earns overall scores lower than a 5, he might ask to see the written assessments.

“If I feel like they’re worth giving another chance, I’ll try to use the data and narrative as one method of trying to do better the next time,” Blair said.

“I think it’s a good tool to allow students to vent. When I read through my own, you can tell who my venters are, and those who have something thoughtful to say… I get lots of suggestions to improve every semester and many semesters I will get one that I act upon.”

Evaluations that get results
In the Producing for Film and Video program, A.G. Akturk, 30, said that she and her classmates feel that the administration has taken their evaluations seriously. She said that, after one course, the class rated the professor poorly and reached out to Blair to complain. The professor was not asked to return, according to Blair.

“It had been a new instructor and it just didn’t work out,” he wrote in an email. “In the weekend program we have mostly adult learners with successful professional careers and their input is taken very seriously.”

Another time, a screenwriting professor told Akturk’s class that he had read the previous cohort’s evaluations and, based on the criticisms, threw out a character-building exercise because it took too long and wasn’t effective.
“The department takes the evaluations to heart and it makes us feel good because it shows they are listening to us,” said Akturk, a freelance writer, film producer and social media consultant who is graduating this spring.

Disgruntled students
At the School of Communication, Robertson said she always keeps in mind that a few students who are upset about their grades can easily influence a professor’s scores. In the written evaluations, students sometimes make irrelevant or even cruel remarks.

“I’ve seen evaluations where you have 15 students who say this is the best professor and one person saying this is the worst instructor,” she said.

In general, she said that anytime a professor scores below a 5 or 5.5 on certain questions, red flags can be raised. But sometimes, administrators take into account that those low scores are payback from students angry over bad grades; or that the class is new or hard.

“Faculty who set high standards in grading are often punished by students in their evaluation,” she said.

But many students such as Rodrigues, the Public Communication student who also works at the university, try being as thoughtful and earnest as possible in their evaluations. And when the evaluations were not enough, some of Rodrigues’ classmates in the program took initiative: They reached out to Hayes, the program director, when they encountered a professor they strongly disliked.

“Many of us discussed our feelings and asked, ‘Did we really got anything out of this?” said Rodrigues, who said she earned As and Bs throughout the program. “So, my classmates wrote him directly about specific professors, feeling like this isn’t what we signed up for, and looking forward, we hope you wouldn’t hire this person anymore. He said he would look into it…It’s good to know our concerns were heard, but I am not sure how much further it goes.”