Real-Time AI-Based ACR TI-RADS Classification of Thyroid Nodules on the Ultrasound Console: A Prospective Agreement Study in a Region with Historical Iodine Deficiency

Background/Objectives: To assess the agreement between an artificial intelligence (AI)-based decision support system (DS System), applied in real time directly on the ultrasound console, and an expert panel for ACR TI-RADS classification of thyroid nodules (TNs) in a region with a history of iodine deficiency and a consequently high prevalence of TNs. Methods: A single-center prospective study including 118 consecutive patients (with 276 TNs) referred for thyroid ultrasound between March and May 2024. DS System ultrasound features, ACR TI-RADS classifications (TR1–TR5) and management recommendations were recorded per TN, with and without the ‘AI-Adapter’ (AI-based modification of the TI-RADS score), and compared to the consensus assessment of a TI-RADS expert panel (paired tests of marginal homogeneity, Cohen’s unweighted and weighted κ; all analyses adjusted for clustering of TNs within patients). Results: No malignancy was identified during 24-month follow-up (composite reference standard; cytology or histology in six TNs). The DS System could be applied to all TNs without technical failure. Marginal distributions differed between the expert panel and the DS System for all ultrasound feature categories except echogenicity (p = 0.23). Interrater agreement ranged from poor (“lobulated or irregular”, κ = −0.08) to substantial (“cystic or almost completely cystic”, κ = 0.62), most frequently slight to fair. TI-RADS classifications showed slight agreement with (κ = 0.04; linear-weighted κ = 0.14) and without AI-Adapter (κ = 0.13; linear-weighted κ = 0.21). The AI-Adapter changed the ACR TI-RADS score by −1.01 ± 1.04 points. Management recommendations showed fair agreement with (κ = 0.23) and without the AI-Adapter (κ = 0.39). Although the number of TR5 ratings was similar (expert panel: 27; DS System: 26 without and 30 with the AI-Adapter), only seven and eight of the 27 TR5 TNs of the expert panel, respectively, were also rated TR5 by the DS System. Conclusions: Real-time application of the DS System on the ultrasound console was technically feasible, but agreement with an expert panel was low. As all TNs were benign, diagnostic accuracy and clinical safety could not be assessed. Validation in cohorts including malignant TNs is required before the DS System can be considered a substitute for expert assessment in regions with a high prevalence of TNs.