Human–AI interaction in parotid ultrasound diagnosis: a prospective two-phase observer-performance study

Differentiating pleomorphic adenoma (PA) from Warthin’s tumor (WT) — the two most common benign parotid neoplasms — on ultrasound is challenging even for experienced ultrasonographers. Most existing AI systems for parotid ultrasound have been static, retrospective, and tested without prospective reader evaluation. We developed a confidence-aware AI system with a radiologist-editable interface and prospectively evaluated its impact on radiologist performance in an observer-performance study. We developed a confidence-aware hybrid pipeline combining a YOLO detection model with a classification model fallback for parotid ultrasound differentiation, using an internal dataset of 949 images (Korea University Ansan Hospital) and an external dataset of 427 images (Korea University Anam Hospital). We exploratorily evaluated 35 detection × classification combinations to characterize model complementarity, and then prospectively validated a single pre-specified deployment configuration (τ = 0.80, YOLOv12m + ConvNeXt) through a two-phase observer-performance study with three board-certified radiologists on 107 external cases (321 reads total). Standalone detection and classification models achieved external AUROCs of 0.811 and 0.919, respectively. The confidence-aware pipeline outperformed both standalone models in 22 of 35 combinations on external data (highest macro-F1 = 0.838) with the complementary effect most apparent when the two models performed comparably. In the reader study, AI assistance improved pooled diagnostic accuracy from 76.3% to 81.6% (generalized linear mixed model [GLMM] odds ratio = 1.54, 95% CI: 1.12–2.13, p = 0.008) and inter-reader agreement (Fleiss’ κ increasing from 0.475 to 0.611), with larger gains on more difficult cases. Reader response was selective: readers followed correct AI predictions more often than incorrect ones (76.2% vs. 51.1%, p = 0.008), and per-class sensitivity changed in opposite directions for WT (+ 18.1% points) and PA (− 9.3% points), aligning with the AI’s prediction profile. A confidence-aware AI system with a radiologist-editable interface improved radiologist diagnostic accuracy and inter-reader agreement in a prospective observer-performance study. These findings support the system’s potential as a decision-support tool that operates alongside the interpreting physician, and motivate further evaluation in real-world clinical workflows.