Deep learning method for detecting fluorescence spots in cancer diagnostics via fluorescence in situ hybridization

Fluorescence in Situ Hybridization (FISH) is a technique for macromolecule identification that utilizes the complementarity of DNA or DNA/RNA double strands. Probes, crafted from selected DNA strands tagged with fluorophore-coupled nucleotides, hybridize to complementary sequences within the cells and tissues under examination. These are subsequently visualized through fluorescence microscopy or imaging systems. However, the vast number of cells and disorganized nucleic acid sequences in FISH images present significant challenges. The manual processing and analysis of these images are not only time-consuming but also prone to human error due to visual fatigue. To overcome these challenges, we propose the integration of medical imaging with deep learning to develop an automated detection system for FISH images. This system features an algorithm capable of quickly detecting fluorescent spots and capturing their coordinates, which is crucial for evaluating cellular characteristics in cancer diagnosis. Traditional models struggle with the small size, low resolution, and noise prevalent in fluorescent points, leading to significant performance declines. This paper offers a detailed examination of these issues, providing insights into why traditional models falter. Comparative tests between the YOLO series models and our proposed method affirm the superior accuracy of our approach in identifying fluorescent dots in FISH images.