Relationships between numerical score and free text comments in student evaluations of teaching: A sentiment topic analysis reveals the influence of gender and culture

Student evaluations of teaching (SET) have been widely used by university staff to inform decisions on hiring and promotion. In recent years, an increasing body of research has revealed that student evaluations may be systemically affected by students' own conscious or unconscious biases. In this article, we study a data set from an Australian university, where both numerical and text survey responses were available in large quantities. Our study directly linked comments to numerical ratings, we developed approaches to convert text to quantitative data in the form of topics and sentiment scores, and make use of Bayesian ordinal regression techniques to identify drivers of SET scores. Our analysis of text identified 6 teaching dimensions that students discuss in their comments. Our findings suggest that students' SET ratings were correlated primarily with the personal characteristics of the lecturer (such as approachability, and being nice) than measures related to teaching dimensions such as course content and assessment. We found a positive gender effect towards the majority gender in a faculty, possibly reflecting students' gendered expectations. Finally we found that lecturers with a non-English language background were consistently rated lower by the student population, and this effect manifests strongly in local students.