EvaluaTUM

Motivation

It might have seemed obvious that all lectures generally required ongoing enhancement. This applied even to the best lectures. The Technical University of Munich was often reputed to be top-notch - not only in the domain of research but also teaching. This superior teaching called for assessment - continuous, unparalleled evaluation. Given all of TUM's brilliance, one might have questioned why their professors - the most intellectually gifted and well-educated individuals - required evaluation in teaching?

Unquestionably, professors had been teaching at TUM for the past 150 years. Equally clear, not all university lecturers were professors. Many lecturers were doctoral students and therefore at the outset of their careers. This signified that they were still eager and required to hone their teaching skills. Even though professors were seasoned lecturers and highly educated, there were reasons why continuous evaluation was imperative.

A professor's role comprised predominantly two elements: research and teaching. Nonetheless, most perceived themselves principally as scientists - not as lecturers - thereby prioritizing the former task. Hence, the necessity for lecturers' training was undeniable. Regrettably, some lecturers disregarded the evaluation process. Thankfully, they represented a minority. Their perspective was: "I am the expert, so I know the best of what and how to teach." In this statement, they overlooked several points. Primarily, many students didn't grasp the content as rapidly as professors. Consequently, the students' input was crucial. Since no one is perfect, there was always a room for improvement, particularly in teaching students. Perhaps the most significant point had nothing to do with the lecturers themselves. In contemporary times, everything evolved swiftly. New teaching strategies emerged rapidly but not every strategy suited every teaching content. As students attended numerous lectures, they were exposed to a plethora of different teaching approaches daily. Hence, they could provide invaluable feedback. Additionally, not only the teaching methods transformed, but also the contents. From this, it was clear that even the finest lecture concept had to be perpetually improved and adapted. Especially, at a so-called superior university. Therefore, it constantly required exceptional feedback and evaluation.

Conversely, the feedback rates at TUM indicated that there was still scope for advancement in the area of lecture evaluation. The members of team EvaluaTUM attempted to address this potential in their research.

Method

First, we conducted an extensive situation analysis for which student associations, lecturers, professors, as well as authorities (e.g., HRSL, ProLehre) were consulted. To implement (new) methods at TUM, they needed to be adapted to meet the TUM guidelines.

We were happy to announce that we had found two partner faculties to help us with our project the following winter semester 2019/20. We accompanied MSE with their change to online evaluation and investigated the impact on the response rate as well as possibilities to overcome potential negative effects. Moreover, we worked together with the chemistry faculty which used paper-based evaluation. By optimizing the space used, shortening, and accentuating the free response questions, we examined the effects on the quality of feedback received by the lecturer.

Current status

We have finalized our results and presented them at the Symposium and in the Research Report.

Results

In the pursuit of superior teaching quality, the necessity for a consistently refined lecture assessment was recognized. Consequently, the Technical University of Munich (TUM) formulated an intricate appraisal system within their quality management domain.

Observing the existing system, this report indicated that minor adjustments in certain variables could enhance the response rate and the caliber of feedback in lecture evaluations.

The analyzed independent variables included the time frame provided to students for questionnaire completion during lectures, the preferred evaluation medium (online vs. paper-based), and an alteration of the questionnaire in terms of length and type of questions.

To validate the three hypotheses mentioned below, three case studies were implemented:

1. Allocating time for students to complete the questionnaire in the midst of the lecture significantly improved feedback rates, as opposed to not motivating the students during the lecture.
2. Compared to paper-based evaluations, online evaluations boosted the response rate when conducted similarly.
3. A compact evaluation questionnaire emphasizing open questions garnered more feedback than a questionnaire concentrating on single-choice questions.

The data that was collected firstly indicated that the lecture's midpoint seemed to be the optimal time for maximizing the feedback rate. Secondly, the online-based evaluation didn't significantly influence the response rates, even though a larger student body was reached. That is, students who couldn't attend the lecture that specific day also had the opportunity to participate. Lastly, a shortened questionnaire with a focus on open questions enhanced the quality of feedback.

From the last two points, it could be inferred that minor adjustments in TUM's current lecture evaluation system could lead to significantly improved lecture evaluations. Several TUM departments have already expressed their interest in our results to upgrade the evaluation system. Owing to this project, one faculty initiated the process of reevaluating their evaluation sheets.