Do Different Rating Scales in Student Evaluations Bias Perceptions of Teaching Quality?

This study examines how different rating scales in Student Evaluation of Teaching (SET) influence perceived teaching quality and decision-making. In Phase 1, SET data was collected from five university courses, with students randomly assigned to evaluate their instructor using one of two different rating scales per course. In Phase 2, 3,719 online participants completed one of two decision-making tasks based on SET results from Phase 1. Participants were randomly assigned to select a university for their studies or hire a candidate, with ratings derived from varying rating scales. A linear regression analysis shows that differences in standardized scores derived from the usage of different rating scales significantly influence selection outcomes. Additionally, a thematic analysis of open-ended responses reveals that participants relied on percentage-based comparisons, but their choices were adjusted when the distribution of responses was provided. These findings uncover the bias introduced by different rating scales in SET.