Training AI to Improve Distinction of Triple-Negative Invasive Breast Cancer from Cysts and Fibroadenomas on Ultrasound

Background/Objectives: Circumscribed oval, hypoechoic masses are common on screening breast ultrasound (US), and the vast majority are benign. Triple-receptor negative invasive breast cancer (TNBC) can appear similar, resulting in both human and artificial intelligence (AI) interpretive errors. Purpose: We sought to improve AI performance in distinguishing common benign masses from TNBC through a retrospective model refinement and validation study. Materials and Methods: In an Institutional Review Board-approved HIPAA-compliant protocol, from five academic medical centers, orthogonal ultrasound images of 1771 breast masses 2 cm or smaller were acquired, consisting of cysts, complicated cysts, other benign, and malignancies. Cases were randomized, controlling for lesion class, site, and patient age, with 1446 (including 402, 27.8%, malignancies) used for training and 325 (including 95, 29.2% malignancies) for validation using Koios DS® (decision support, KDS) software version 2.0. A breast imaging radiologist from each center reviewed images and recorded BI-RADS features and assessment. Demographics, symptoms, and pathology or at least one-year follow-up was recorded. The KDS score was evaluated standalone and in combination with BI-RADS using logistic regression and ROC analysis with focus on specificity at sensitivity of 98%. Results: In training, KDS standalone performed comparably to BI-RADS, and significantly improved BI-RADS malignancy risk prediction (p < 0.001). The 98%–sensitivity threshold for combined KDS + BI-RADS was estimated and kept fixed during validation. In validation, KDS standalone performed similar to BI-RADS with AUC = 0.97 (CI: 0.95–0.98) versus 0.95 (p = 0.22), with sensitivity of 98% (93/95, CI: 95–100%) for both and specificity of 70.9% (163/230, CI: 65.0–76.7%) for KDS versus 63.9% for BI-RADS (147/230, p = 0.10). Combining KDS + BIRADS significantly improved overall performance (AUC 0.98, p < 0.001) and specificity (74.4%, 171/230, p < 0.001) while maintaining sensitivity at 98% (93/95). Conclusions: While KDS alone should not replace BI-RADS, when used in combination with BI-RADS, it can significantly improve specificity for highly accurate (98% sensitivity) triaging management of masses representative of those seen on screening US.