Evaluating the Accuracy and Reliability of StudyMate AI for Document-Based Question Answering and Multiple-Choice Question Generation

This study evaluates the accuracy and reliability of StudyMate AI, an educational AI system designed to answer questions and generate multiple-choice questions from student-provided educational materials. The evaluation used three educational documents covering Physics, Chemistry, and Computer Science. Two controlled experiments were conducted. The first evaluated document-based question answering using nine predefined questions and reference answers derived from the source materials. The second evaluated multiple-choice question generation using 45 generated questions across three subjects and three requested difficulty levels. StudyMate correctly answered 8 of 9 question-answering questions (88.9%). The only incorrect response occurred in a Hard Physics question involving an isochoric process, where Celsius temperatures were used directly instead of being converted to Kelvin. For multiple-choice question generation, all 45 generated questions were evaluated as content-correct, source-grounded, and having unique correct answers under the predefined evaluation criteria. The repository contains the research paper and the evaluation datasets used in the controlled experiments. The source educational materials are not included in this record.