Departments debate efficacy of online, paper course evaluations

The number of sheets of paper that the Office of Institutional Research consumed to conduct course evaluations last spring is 25,842.

With only 22 percent of Lehigh faculty opting for the online evaluation form, Institutional Research is left with the lengthy and environmentally-unfriendly process of mass producing the forms and vibrant orange envelopes that land on professors’ podiums at the end of each semester. This process has created a debate between the use of online versus paper course evaluations.

The Office of Institutional Research Coordinator Debbie Hamann said each semester, the office begins around the middle of a term by extracting course information including course title, enrollment and instructor, and sending out preliminary listings to department coordinators to confirm their accuracy. After the listings are finalized, they are sent back to the office for a three-week printing process. Approximately 1,500 courses are evaluated every fall and spring semester. Many courses have various teaching assistants and sections, which contributes to the large number of packets.

(Kelly McCoy/Made with Illustrator)

Faculty members that use paper generally believe they receive better response rates because they can administer the evaluation forms in person to students who are already sitting at a desk ready to fill them out. Students, however, are left feeling pressed for time and may not provide as thorough feedback as if they took it online at a different time.

Katherine Ries, ’16, is taking a course called Science of Environmental Issues, which has several online course evaluations throughout the semester because various professors teach the class in short modules on environmental issues.

“Handing out evaluations in class ensures that all students will fill out the course evaluation,” Ries said. “However, I often feel like I’m being rushed to fill it out at the end of class and don’t put much thought into it. Online evaluations give students the opportunity to fill out the evaluation on their own time, but I think it is less likely that students will actually complete it.”

Jim Dearden, chair of the department of economics, said the department has tried voluntary online evaluations in the past and found that response rates are lower and the distribution of scores are more widespread. He said results tend to have selection bias since students that have strong opinions are more likely to complete the evaluation than more neutral students.

“I would love to have online evaluation forms, but they would need to be mandatory in order to avoid bias,” Dearden said. “If they are voluntary, then they aren’t indicative of the course.”

As a result of this bias, it’s difficult to compare professors who use online versus paper forms.

He suggested making online evaluations mandatory by only permitting students to view their final grades once they have completed the evaluation. This obligatory online format would save time in the classroom and reach more students because, oftentimes, not all the students in the course show up to class the day the evaluation forms are handed out anyway.

Reis said this mandatory method seems extreme, and it’s probably best to do them on paper in the classroom if evaluations need to be completed.

Yenny Anderson, the director and interim vice provost for Institutional Research, said one drawback to using paper is that the office cannot monitor and ensure the proper procedure is followed for administering the evaluations in classrooms.

The professor is supposed to read the official instructions, select a student volunteer to distribute and collect the forms, and then allow a full 15 minutes for the students to complete the surveys. However, there is no way to make sure it’s done correctly and fairly.

Sometimes the packet of forms never makes it back to the office if a student does not follow through with the instructions or forgot to return the envelope. The office loses control of the packet once it is sent out to the departments, so only about 75 percent actually come back for full processing.

The Office of Institutional Research believes conversion to the online evaluation format is an environmentally friendly solution that can actually improve the likelihood for more feedback and better quality responses. Currently, the College of Education and English as a second language department are the only groups to completely administer evaluations online.

This fall semester, the Office of Institutional Research is testing out new software, EvaluationKIT, which had promising results when it was piloted this past summer session, Anderson said. The new program makes more information available to faculty and departments and eases the evaluation process as a whole.

For example, students can receive a link to a portal that houses all their online course evaluations. It also connects to Lehigh’s Coursesite so students will be reminded via Coursesite to complete pending evaluations. With EvaluationKIT, faculty can view response rates online and remind students in class to take the evaluation during the two-week evaluation period.

According to the office, in Spring 2015, the general online evaluation response rate was 70.9 percent, which is just a few percentage points below the paper evaluation response rate of 74.7 percent.

“The quality of comments has improved significantly and amount of comments has increased online,” Anderson said. “Students have more time to write out comments and give an opinion or complaint about course that they wouldn’t usually have in a classroom setting. They can really think about their responses and even leave and come back to the computer later to finish if they want.”

Another advantage to the online format is that the summaries can be viewed shortly after grades are finalized instead of waiting weeks for feedback, which is how long it takes when paper evaluations have to be scanned and processed manually.

“A conversation about a greater push towards online evaluations has been had but it was determined that (the office) is solely in charge of administering the evaluations. It is unclear who has the authority to implement that type of change,” Anderson said.

She also said as faculty become more conscious of being green, more instructors will ideally opt into the online format. Turnover in faculty and a push from students may also help move Lehigh toward paperless course evaluations.

In terms of what actually happens with the results of course evaluations is a question that lingers in students’ minds. Once the evaluations are processed by the Office of Institutional Research, the feedback for each instructor and course is sent back to the departments who share results with instructors after final grades have been posted.

“(The office) provides data uploads to a person in each of the four colleges,” Anderson said. “The person uploads the data file containing the evaluation results for each faculty into a system that collects information to be used for faculty annual review. In essence, the evaluation results are incorporated into the material used to review faculty overall performance.”

For the economics department, Dearden said that the course evaluations are taken into consideration for tenure and performance evaluations but they are just one piece of information used in evaluating a faculty member.

Other factors include in classroom observations by peer faculty members as well as course test scores. Dearden said student evaluations generally indicate if a professor or teaching assistant is struggling and needs help improving their teaching style.