Evaluating AI-mediated Tutors for Competitive-Programming Training: A Position Paper on Prompt-Family Ablation and Rubric Design
Large language models are routinely deployed as tutors in introductory programming (CS1/CS2), and an evaluation literature has emerged around that surface. Competitiveprogramming (CP) olympiad preparation, in contrast, presents a qualitatively different tutoring problem: multi-step algorithmic composition, tight complexity budgets, and adversarial problem framing. No peer-reviewed study evaluates LLM tutors against CP olympiad learners or against a milestone-differentiated cognitive model of CP skill acquisition. This position paper makes three contributions. First, it argues that the CP-tutor evaluation slot is structurally distinct from the CS1-tutor evaluation slot and cannot be filled by extrapolation. Second, it proposes a five-axis rubric (correctness, pedagogical fit, transcript coherence, bias behaviour, Bloom alignment) that extends the four-axis variation schema used in sister work on CP-problem generation. Third, it specifies a three-prompt-family ablation design, one family per knowledge-compilation milestone (L3 strategy retrieval, L4 algorithm adaptation, L5 complexity reasoning), together with a two-channel evidence plan (a Bayesian-Knowledge-Tracing simulated learner and an adult-proxy role-play protocol) that respects a post-pivot admissibility regime excluding K-12 subjects. The paper is positioned as the tutor-side complement to an existing CP-problem-generator methodology, sharing the variation-axis backbone, the pre-registration discipline, and the LLM-as-judge bias-control frame.
Authors
- That Le (ORCID: https://orcid.org/0009-0004-3361-3046)
Institutions
- ElSohly Laboratories (United States) (US)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-05-12
- DOI
- https://doi.org/10.5281/zenodo.20131499
- Primary Topic
- Teaching and Learning Programming
- Type
- preprint