Enhancing AI Feedback Systems: Lessons Learned from a RAG Implementation Study

This study investigates the impact of Retrieval-Augmented Generation (RAG) on graduate students' perceptions of AI-generated feedback in a cyber-social learning environment. Using a non-equivalent cohort comparison quasi-experimental design, the research compared survey responses from Fall 2023 (n=96) and Spring 2024 (n=86) before and after implementing RAG. Participants engaged in iterative project cycles involving AI feedback, peer reviews, and instructor evaluations. Results revealed significant improvements in perceived feedback quality (p < .001, d = 0.41) and usefulness (p < .001, d = 0.51) following RAG implementation, while perceptions of actionability remained unchanged (p = .68, d = -0.05). Hierarchical regression analyses identified educational level as a positive predictor of quality and usefulness perceptions, whereas age and location negatively predicted actionability, and educational level positively predicted it. These findings suggest that RAG enhances the contextual relevance and perceived value of AI-generated feedback, particularly for advanced students. However, the lack of improvement in actionability highlights a persistent challenge in AI feedback systems. The study recommends integrating RAG with human support and developing targeted interventions to improve feedback literacy and implementation, especially for less advanced students and older learners.

Authors

Institutions

Publication Details

Journal
Journal of University Teaching and Learning Practice
Published
2026-11-14
DOI
https://doi.org/10.53761/9vprqq24
Primary Topic
Artificial Intelligence in Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Enhancing AI Feedback Systems: Lessons Learned from a RAG Implementation Study

Akash Kumar Saini, Mary Kalantzis, Bill Cope, Christopher Hughes
Journal of University Teaching and Learning Practice
Artificial Intelligence in Education
article

Enhancing AI Feedback Systems: Lessons Learned from a RAG Implementation Study

Akash Kumar Saini, Mary Kalantzis, Bill Cope, Christopher Hughes
article en

Abstract

This study investigates the impact of Retrieval-Augmented Generation (RAG) on graduate students' perceptions of AI-generated feedback in a cyber-social learning environment. Using a non-equivalent cohort comparison quasi-experimental design, the research compared survey responses from Fall 2023 (n=96) and Spring 2024 (n=86) before and after implementing RAG. Participants engaged in iterative project cycles involving AI feedback, peer reviews, and instructor evaluations. Results revealed significant improvements in perceived feedback quality (p < .001, d = 0.41) and usefulness (p < .001, d = 0.51) following RAG implementation, while perceptions of actionability remained unchanged (p = .68, d = -0.05). Hierarchical regression analyses identified educational level as a positive predictor of quality and usefulness perceptions, whereas age and location negatively predicted actionability, and educational level positively predicted it. These findings suggest that RAG enhances the contextual relevance and perceived value of AI-generated feedback, particularly for advanced students. However, the lack of improvement in actionability highlights a persistent challenge in AI feedback systems. The study recommends integrating RAG with human support and developing targeted interventions to improve feedback literacy and implementation, especially for less advanced students and older learners.

Journal of University Teaching and Learning Practice
University of Illinois Urbana-Champaign (US)
Artificial Intelligence in Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon