GeoChromNet: Declustering Metaphase Chromosomes via Topology-Informed Deep Instance Segmentation

Accurate segmentation and declustering of metaphase chromosomes are crucial for automated karyotyping and diagnosing genetic disorders. However, the complex arrangements in metaphase spreads, including overlapping and touching chromosomes, challenge existing segmentation approaches. This paper presents GeoChromNet (Geometric Chromosome Network), a novel deep learning architecture utilizing topological information to enhance metaphase chromosome instance segmentation and declustering. Our model combines a modified Mask R-CNN backbone with a topology-aware module to effectively segment and decluster chromosomes. We evaluate GeoChromNet on two datasets along with augmentation: The Cell Image Library (CIL), with 5,000 metaphase cell images and 229,852 annotations across 24 chromosome classes, and the BioImage Chromosome Classification Dataset (BICCD), containing 42 chromosome objects with various annotations. Experimental results show GeoChromNet outperforms state-of-the-art methods like ChroSegNet, Mask R-CNN, PSPNet, and DeepLabV3. On the CIL dataset, GeoChromNet achieves a Mean Pixel Accuracy (MPA) of 96.8%, an F1 score of 0.945, a Mean Intersection over Union (MIoU) of 0.921, a Jaccard index of 0.893, and a Dice similarity coefficient of 0.937. These results represent improvements of 2.3%, 3.1%, 3.7%, 4.2%, and 2.8% respectively over the next best model, ChroSegNet. GeoChromNet’s topology-informed approach is particularly effective in declustering challenging cases of overlapping and touching chromosomes, as evidenced by a 15.% increase in cluster separation accuracy compared to ChroSegNet. On the BICCD dataset, GeoChromNet maintains its superior performance, achieving an MPA of 95.7% and an F1 score of 0.932, demonstrating its robustness across different datasets. Our work highlights the potential of integrating topological information into deep learning models to improve the accuracy and robustness of metaphase chromosome segmentation and declustering, paving the way for more reliable automated karyotyping systems.