Pedagogy-in-the-Loop: An Efficient Taxonomy-Grounded Classifier for Real-Time AI Tutor Evaluation

Despitethe considerable potential of large language models (LLMs) as interactive AI tutors, the systematic, large-scale evaluation of their pedagogical quality remains an open challenge. Assessment instruments currently in use are mostly based on informal criteria, or task completion is based on ad hoc criteria or binary task-completion indicators. In response, we propose a lightweight and edge-deployable classifier based on the MRBench taxonomy, which applies the principle of experts teaching and explicitly rates the tutor’s responses for guidance, actionability, and coherence. The architecture is embedded with training through rigorous Stratified Group K-Fold Cross-Validation, which avoids semantic leakage from the augmented datasets and is backed by an XGBoost model with TF-IDF vectorization. To ground statistical predictions in pedagogical theory, we add a deterministic rule layer that preserves canonical scaffolding interventions. This hybrid classifier achieves a strict out-of-fold (OOF) accuracy of 68.2% and an F1-score of 0.74. The deterministic rule layer acts as a qualitative safety net and does not statistically change the overall shift (McNemar’s test, p=0.25), but still helps to identify pedagogical moves that are valid, while maintaining predictive fidelity. Most importantly, single-threaded edge-simulated benchmarking of the complete end-to-end pipeline (vectorization and classification) shows a mean latency of 116.20 ms, compared to an average of 6834 ms per inference for a locally hosted 3B-parameter LLM baseline. In conclusion, this transparent and taxonomy-based framework provides a practical, real-time auditing toolkit for educators and researchers to benchmark AI tutors and observe model behavior in resource-limited deployment environments. The proposed lightweight framework ensures that robust, taxonomy-grounded tutor evaluation can be carried out without relying on costly, cloud-based LLM-as-a-judge pipelines, allowing for real-time pedagogical auditing of AI tutors in resource-limited environments.