When Socratic Tutoring Scores Track Prompts: Sensitivity Analysis of Simulated Statistics Dialogues

Language-model tutors can generate Socratic-looking dialogue, but evaluation must determine whether questions are correct, learner-responsive, and educationally meaningful. This study reanalyzes a completed prompt comparison using Qwen3-14B as both tutor and simulated learner. Four tutor conditions—direct explanation, minimal Socratic instruction, fixed Socratic script, and adaptive Socratic prompt—were crossed with three introductory-statistics topics, two prompt phrasings, and three persona-linked seeds, yielding 72 eight-turn dialogues. A mechanically scored Open Socratic Index initially gave the adaptive condition a mean of 63.0 and the fixed script a mean of 62.9, a paired difference of 0.1 points. The adaptive condition appeared to exceed direct explanation by 8.93 points, but 8.67 points came from a transfer detector that rewarded wording mandated by the adaptive treatment. Removing that detector reduced the contrast to 0.26 points, with an exact sign-flip p-value of 0.935. The protocol also gave tutors full numerical context while withholding it from simulated learners, so content accuracy could not be compared fairly across conditions. In this corpus, prompt rankings reflected information access and metric construction rather than demonstrated pedagogical adaptivity. The contribution is a diagnostic workflow for decomposing composite tutoring scores, testing treatment-independent sensitivity, and specifying evidence needed before simulated dialogues support educational claims.