Proactive framework for evaluating retrieval‐augmented generation‐based learning assistants in engineering education

Retrieval-augmented generation (RAG) enabled learning assistants are promising for engineering education, given their capability to supplement domain-specific knowledge and enhance student support. However, it is also a known problem that RAG demands adequate knowledge bases and can experience unreliable retrieval generation alignment. This study proposes a proactive evaluation framework for RAG-based learning assistants, eliminating the need for student feedback in system evaluation. The framework is demonstrated using a Civil Engineering education tool, CivASK. The evaluation framework identifies the deficiencies in CivASK, including database gap, contextual misunderstanding, and incomplete retrievals, based on the performances under simulated student inquiries, automated retrieval ranking, and expert-validated evaluations. Specifically, 742 student queries are analyzed, and 374 test questions are generated for assessment, showing the practical utility of the proposed evaluation framework for real-world education assist development. The application of the proposed framework is transferable to assist other engineering courses as well.