Training-corpus overlap and evaluation protocol constrain measured gains in a fine-tuned clinical language model

Abstract Fine-tuned medical language models are frequently assessed using public question-answering benchmarks, which may overlap with their training data. We re-evaluated an 8.2-billion-parameter QLoRA fine-tune of Qwen3-8B under a fixed generative chain-of-thought harness and compared eight benchmarks against its 97,041-item fine-tuning corpus using 13-gram, exact-stem and longest-common-subsequence matching. Exact-stem overlap exceeded 70% for PubMedQA, MedQA and MMLU Professional Medicine, making those scores unsuitable for capability claims, while MedMCQA, evaluated on a held-out split, was effectively unflagged. On several overlapping benchmarks the unmodified parent checkpoint matched or exceeded the fine-tune, significantly so on MedQA. On the 1,838 MedXpertQA items not flagged by exact-stem matching, the fine-tune scored 20.8% against 15.6% for its parent, a paired gain of 5.2 percentage points (McNemar exact P = 7.9 × 10⁻⁷), against an originally reported 7.1. A bundled ablation, in which reasoning traces, sampling weights, curriculum staging, token exposure and output-format supervision differed simultaneously, preserved approximately 70% of the observed improvement. The remaining 1.6-point difference was not statistically significant ( P = 0.175) and cannot be attributed to any single component. The evaluation protocol and the overlap of the training corpus limit the scope of measurable outcomes in these benchmarks.