Scalable Evaluation of Retrieval-Augmented Generation Systems for Educational Question Answering

AI tutors powered by Retrieval-Augmented Generation (RAG) are increasingly used in education, but their reliability depends on retrieving appropriate instructional content. This study quantifies RAG performance in an educational setting using benchmarks of varying complexity derived from a biotechnology curriculum. We began with a baseline RAG system evaluated on a manually constructed question–answer benchmark, then introduced automated benchmark generation to scale evaluation across RAG configurations and identify the most impactful design choices. We further evaluated the best-performing configuration on textbook-section benchmarks labeled by Bloom's Taxonomy and Webb's Depth of Knowledge. Results show that architectural choices such as retrieval strategy and chunking were associated with larger performance differences than parameter-level adjustments, with Milvus hybrid retrieval achieving higher success rates than Pinecone dense retrieval in this benchmark. Full-textbook retrieval also improved performance on some higher-difficulty questions compared to section-limited search. Together, these findings underscore the importance of systematically evaluating retrieval reliability for AI tutors in complex biotechnology domains.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-17
DOI
https://doi.org/10.5281/zenodo.24323516
Primary Topic
Intelligent Tutoring Systems and Adaptive Learning
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Scalable Evaluation of Retrieval-Augmented Generation Systems for Educational Question Answering

Dylan Bulseco, Andriy Kashyrskyy, Aarman Jamiluddin
2 links
Zenodo (CERN European Organization for Nuclear Research)
Intelligent Tutoring Systems and Adaptive Learning
article

Scalable Evaluation of Retrieval-Augmented Generation Systems for Educational Question Answering

Dylan Bulseco, Andriy Kashyrskyy, Aarman Jamiluddin
article en

Abstract

AI tutors powered by Retrieval-Augmented Generation (RAG) are increasingly used in education, but their reliability depends on retrieving appropriate instructional content. This study quantifies RAG performance in an educational setting using benchmarks of varying complexity derived from a biotechnology curriculum. We began with a baseline RAG system evaluated on a manually constructed question–answer benchmark, then introduced automated benchmark generation to scale evaluation across RAG configurations and identify the most impactful design choices. We further evaluated the best-performing configuration on textbook-section benchmarks labeled by Bloom's Taxonomy and Webb's Depth of Knowledge. Results show that architectural choices such as retrieval strategy and chunking were associated with larger performance differences than parameter-level adjustments, with Milvus hybrid retrieval achieving higher success rates than Pinecone dense retrieval in this benchmark. Full-textbook retrieval also improved performance on some higher-difficulty questions compared to section-limited search. Together, these findings underscore the importance of systematically evaluating retrieval reliability for AI tutors in complex biotechnology domains.

Zenodo (CERN European Organization for Nuclear Research)
Institute for Future Intelligence (US)
National Science Foundation
Quality education
Intelligent Tutoring Systems and Adaptive Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon