Comparative Evaluation of YOLO Architectures for Automated Breast Cancer Detection in Digital Mammography

Early detection of breast cancer is critical for improving patient survival; however, accurate interpretation of digital mammography remains challenging due to dense breast tissue, overlapping anatomical structures, and low-contrast lesions. Recent advances in deep learning particularly object detection frameworks from the YOLO family; have shown promise for automated lesion detection. Nevertheless, systematic and controlled comparisons of contemporary YOLO architectures in mammography remain limited. This study presents a rigorous comparative evaluation of three lightweight YOLO variants YOLOv5n, YOLOv8n and YOLOv11n for automated breast lesion detection in digital mammography. Experiments were conducted on the VinDr-Mammo dataset comprising over 20,000 expert-annotated mammograms. To ensure fairness and reproducibility, all models were trained under identical conditions using a clinically validated preprocessing pipeline, including Contrast_imited_adaptive histogram equalization (CLAHE), bilateral filtering and safety preserving data augmentation. A patient wise five-fold cross validation strategy was employed. Model performance was assessed using lesion level metrics including mean average precision, precision, recall, and F1-score, alongside image level receiver operating characteristic and precision–recall analyses. While all models showed good performance, YOLOv11n outperformed all the other models, attaining an mAP₀.₅ of 68.28% and mAP₀.₅:₀.₉₅ of 40.82%, which is 5.4% and 9.2% better than YOLOv8n and YOLOv5n, respectively. YOLOv11n also displayed superior performance metrics, achieving better sensitivity (0.69) and precision (0.73) especially on small and slightly contrast lesions, all while achieving real-time performance at 92 fps and lower GPU memory usage. The results represent the best available performance and document an increase in accuracy and efficiency for the clinically actionable AI-based CAD systems.