Do student perceptions mirror reality? A mixed-methods study to evaluate the accuracy, utility and efficiency of AI-assisted feedback in introductory mathematics courses
Abstract Background Giving meaningful feedback on assignments in large introductory-level mathematics courses is a challenge. The emergence of “out of the box” Large Language Models (e.g., ChatGPT, Gemini) has created a potential solution to this challenge, where the AI tool can be trained to give constructive feedback on student work. However, it is unclear whether these tools are sufficiently accurate, efficient, and trustworthy. The current study investigates the use of a bespoke AI tool to assist in giving feedback to students in two introductory mathematics courses that enroll over 400 students each and are characterized by high attrition. Results We evaluated the similarities in grades assigned by the AI tool and human graders, as well as student and grader perceptions of the feedback given by both the AI tool and human graders. We found that the tool was fairly accurate, but that perceptions of the utility of its feedback varied by condition. When evaluated as a stand-alone grading agent, the AI tool was significantly undervalued relative to human graders—both student peers and the professor. Conclusion Three themes that characterize our findings: student trust is key, the professor is irreplaceable, and graders and AI tools must be carefully trained to ensure accuracy, transparency, and trust. We conclude with a call to exercise caution in using these tools in grading assistance, particularly with respect to maintaining student trust and transparency in grading.
Authors
- Antonella Zanna Munthe-Kaas (ORCID: https://orcid.org/0000-0001-7087-3059)
- Christian Bru
- Therese Romslo Saltskår
- Sehoya Cotner
- Kasper Troøyen
Institutions
- University of Bergen (NO)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-14
- DOI
- https://doi.org/10.21203/rs.3.rs-11455953/v1
- Primary Topic
- Artificial Intelligence in Education
- Type
- preprint