Methodological variation in studying staff and student perceptions of AI

Introduction In this paper, we compare methodological approaches for comparing student and staff perceptions, and ask: how much do these measures vary across different approaches? We focus on the case of AI perceptions, which are generally assessed via a single quantitative or qualitative measure, or with a mixed methods approach that compares two distinct data sources — e.g., a quantitative questionnaire with qualitative comments. Methods To compare different approaches, we collect two forms of qualitative data: standalone comments and structured focus groups. We conduct two analyses for each data source: with a sentiment and stance analysis, we measure overall negativity/positivity of the comments and focus group conversations, respectively. Meanwhile, word clouds from the comments and a thematic analysis of the focus groups provide further detail on the content of this qualitative data — particularly the thematic analysis, which includes both similarities and differences between students and staff. Results We show that different analyses can produce different results — for a single data source. This variation stems from the construct being evaluated — an overall measure of positivity/negativity can produce a different picture from more detailed content-based analyses. Discussion We discuss the implications of this variation for institutional contexts, and for the comparisons from previous studies.