Self-selection bias in the sensitive topic of teacher grading: the role of subject and school context

Self-selection bias in voluntary questionnaire surveys poses a threat to the validity of research findings, particularly when studies address sensitive topics. This paper reports on an unexpected case of self-selection bias encountered in research on teachers’ grading beliefs and practices in Austria. Originally designed to examine the relationship between teachers’ grading beliefs and their actual grading behavior, the study revealed systematic differences between respondents and nonrespondents. Drawing on grade data from 154 teachers, linked with teacher questionnaire responses (n = 56) and nonresponses (n = 98), we found that survey respondents showed significantly different grading patterns compared to nonrespondents. Furthermore, school type (academic vs. vocational upper secondary schools) emerged as a key moderating factor. A subsequent qualitative study with 14 teachers from the nonresponding subgroups (mathematics teachers from both academic and technically orientated vocational schools) provided further insights into the mechanisms underlying this bias, including perceived low value of research topic for nonrespondents, school type specific teacher mentality, reluctance to disclose potentially questionable grading practices, and professional reputation concerns. By documenting this finding, the paper highlights the role of institutional factors (i.e., school type) and subject area in self-selection bias and emphasizes the need for more explicit consideration of such bias in research on teachers, especially when addressing sensitive topics like grading.