Automated detection of HER2 gene copy number in breast cancer using deep learning techniques

Accurate evaluation of HER2 gene amplification is critical for guiding breast cancer treatment decisions. This study proposes a deep learning-based diagnostic system for analyzing Dual In Situ Hybridization (DISH) images to support HER2 status assessment. The system integrates two models— YOLOv11-seg for cell detection and and YOLOv11 object detection models for HER2 and CEP17 signal quantification—into a unified pipeline. High-resolution whole-slide images were preprocessed and annotated to train the models, which were then embedded into a standalone application designed for clinical environments. Upon uploading TIFF format images, the application performs automated cell detection, red/black signal analysis, and HER2/CEP17 ratio computation. Experimental results demonstrated an accuracy 95.24% for the best identification and mean deviations of 6.08% (CEP17) and 12.78% (HER2) compared with manual counting. Statistical analyses confirm high consistency, particularly in red signal detection. Clinical feedback under scores the system’s ease of use, accuracy, and potential to reduce diagnostic burden. The proposed approach demonstrates strong feasibility for routine adoption in pathology workflows. • YOLOv11seg-based model achieved a 96.77% success rate in detecting nuclei from HER2 DISH images. • Automated quantification showed mean deviations of 6.08% (CEP17) and 12.78% (HER2) compared with manual counting. • Developed a standalone clinical application for TIFF image upload, automated analysis, and visualized reporting. • Clinical validation by pathologists confirmed reduced workload and improved consistency in HER2 evaluation.