Case-level audit of thyroid ultrasound and cytology deep learning: label conflicts, split design, and shortcut-sensitive performance in a public archive

The measured performance of thyroid imaging artificial intelligence depends on how the analytical population is assembled, how images are partitioned, whether non-anatomical image content is informative, and which reference labels are used. We audited a public multimodal thyroid archive at case level and examined what its reported performance does and does not demonstrate. The released archive of Pang et al. (387 ultrasound images; 1,332 fine-needle aspiration cytology blocks) was audited for exact and near-duplicate images and for contradictory labels, and a frozen case-level partition was built. Five ImageNet-initialised backbones and their mean-probability ensemble were evaluated by group-stratified five-fold cross-validation and on the frozen test set, with thresholds selected on validation data and case-clustered 95% confidence intervals. Revision-stage control analyses comprised an identical-code comparison of image-level and case-grouped folds; retraining with a heuristically detected sonographic sector or its surround removed, for all five backbones and the ensemble; and a metadata-only model without a convolutional network. Frozen ultrasound models were then reassessed on TN3K, against its distributed benign/malignant labels, and on DDTI, for concordance with TI-RADS categories; both datasets had been accessed during earlier analytical development. Nine ultrasound images appeared under both benign and papillary carcinoma labels (19 files) and were excluded. Under case-level cross-validation, the cytology ensemble reached an AUROC of 0.958 (95% CI 0.938–0.975) and the ultrasound ensemble 0.768 (0.716–0.819). Image-level folds, despite 99.8% within-fold case contamination, changed AUROC by −0.002 (−0.021 to 0.019) and +0.019 (0.008 to 0.032) for the two tested backbones. After removal of the detected sonographic sector, the ultrasound ensemble retained an AUROC of 0.690 (0.630–0.747), a paired change of −0.078 (−0.148 to −0.013); with only the detected sector retained, it reached 0.760 (paired change −0.008, −0.054 to 0.039). Sector detection fell back to a fixed central box in 24.2% of frames. A metadata-only model using frame geometry, global intensity and border statistics reached 0.749 (0.692–0.804) for ultrasound. In post-exposure reassessment, the ensemble reached 0.700 (0.661–0.741; image-level interval) on TN3K and over-predicted malignancy (calibration-in-the-large −0.49), and showed no concordance with DDTI TI-RADS grouping (0.518, 0.436–0.605). This case-level audit of a public thyroid archive showed contradictory labels, partition-sensitive evaluation, and substantial discriminative signal both after removal of a heuristically detected sonographic region and in acquisition metadata alone, so performance on this archive cannot be interpreted as exclusively lesion-based. Post-exposure reassessment on external datasets showed limited and endpoint-dependent transport of the internal models.