Small aggregate accuracy changes conceal substantial item-level correctness turnover after fine-tuning in medical question-answering

Background Comparisons between general-purpose and medically specialised language models are usually made between fixed checkpoints at a single operating point, varying model scale, training data, and inference scaffolding simultaneously, while leaving the fine-tuning configuration uncontrolled. Methods Starting from one 27-billion-parameter parent checkpoint and holding the scale, architecture, evaluation items, decoding, and scoring fixed, we evaluated five fine-tuned derivatives differing in reasoning-trace content, trace length, and training weight. All were evaluated on the identical 2,450-item MedXpertQA-Text test set, providing paired item-level comparisons tested by McNemar exact tests. Results The MedXpertQA-Text test accuracy spanned 16.86 percentage points across the parent and its derivatives (43.76%–26.90%) and 15.30 points among the derivatives themselves. Because configurations differ in several training dimensions and one run was performed for each, these are checkpoint-level differences and do not isolate any single training variable. A range of this magnitude can complicate comparisons between unmatched generalist and specialist checkpoints. No derivative exceeded the parent's accuracy. Aggregate accuracy proved a poor summary of change: Four derivatives finishing within 2.41 points of the parent differed from it on 15.1% to 18.2% of items, and on 12.8% to 14.5% after excluding items on which either model reached the generation cap. Retention rates ranged from 47.2% to 80.3%. Conclusion No consistent descriptive relationship between rationale length and accuracy was observed, and no category-level retention differences survived the multiplicity correction. In conclusion, aggregate accuracy reports the net direction of change after fine-tuning, but not its magnitude, and can conceal substantial redistribution in which questions a model answers correctly. When a parent model and its fine-tuned derivatives are evaluated on identical items, retention, acquisition, and correctness turnover are directly computable and should be reported along with the aggregate accuracy.