When the Validation Set Cannot See: Measurement-Disciplined Development of a Knee-MRI Classifier Supervised by Machine-Read Radiology Reports

Full experimental record of an entry to the RSNA Knee Abnormality Detection Kaggle competition: 4,407 training examinations labelled only by multilingual free-text radiology reports, scored on a hidden test set labelled from scratch by expert readers (12 findings, mean column-wise ROC-AUC). Over seven weeks and about 290 numbered experiments (exp001-exp314, 2026-08-09 to 2026-09-26) we built a 2.5D ConvNeXt pipeline with label-query attention, LLM-extracted soft labels, supervised pretraining on the Osteoarthritis Initiative (OAI), and distillation from fold-blind teachers, reaching 0.950 public leaderboard (rank 52 of 4,315 at 2026-09-25 snapshot). Central claims: (1) Cross-validation against training labels is blind, not noisy, to label-target changes. (2) Ensemble-class gains transfer ~0%, input-class gains at/above prediction (calibration LB = basis + 0.03582). (3) Per-label leaderboard probe (AUCcol = 12*print - 5.5, resolution 0.012) with >=3-step founding rule. (4) Gold-58 subset mis-states Synovitis by ~0.2 and ranks it backwards. (5) Shared-error-floor s~0.047 explains all 8 cross-family blends at 0.948-0.950. ~40 null results with magnitudes + silent notebook-scoring failures documented. Files: paper.pdf = manuscript (30 pages, 9 tables, 4 figures); arxiv_upload.tar.gz = LaTeX source (main.tex + sec_*.tex + refs.tex + figures/) to rebuild it. Abstract: We report the full experimental record of an entry to a knee-MRI Kaggle competition whose structure is becoming common in medical imaging: 4,407 training examinations labelled only by multilingual free-text radiology reports, scored on a hidden test set labelled from scratch by expert readers (12 findings, mean column-wise ROC-AUC). Over seven weeks and about 290 numbered experiments we built a 2.5D ConvNeXt pipeline with label-query attention, LLM-extracted soft labels, supervised pretraining on the Osteoarthritis Initiative, and distillation from fold-blind teachers, reaching 0.950 on the public leaderboard (rank 52 of 4,315 at the last recorded snapshot). Our central claim is methodological. Cross-validation against the training labels is not merely noisy but blind to changes of the training target: a label programme endorsed by three internal instruments lost on the leaderboard. Transfer to the hidden test depends on the type of gain: ensemble-class gains transferred at about 0%, input-class gains at or above prediction. A per-label leaderboard probe turned a rate-limited board into a column-resolved instrument; one column-targeted label change moved the board exactly as priced, while a one-print-step probe advantage did not. After the plateau at 0.950, blends with independent public and in-house model families all printed 0.948-0.950, a one-parameter shared-error-floor model (s ~ 0.047) explains all eight blend prints, and a Qwen3.5 vision-language arm never reached usable strength: report-supervised image models get the same examinations wrong. Image-derived labels from frontier and local readers were screened on 58 expert-labelled examinations; only a paid reader carried signal. We catalogue about forty null results with magnitudes and document silent failure modes of notebook-scored competitions. Competition: https://www.kaggle.com/competitions/rsna-knee-abnormality-detectionCite as: Sikder, A. R. (2026). When the Validation Set Cannot See: Measurement-Disciplined Development of a Knee-MRI Classifier Supervised by Machine-Read Radiology Reports. Zenodo. https://doi.org/10.5281/zenodo.23334291