GMR-4234 Evaluating Instance Segmentation Models on Histopathology Datasets

Instance segmentation is transforming digital pathology by enhancing the speed and accuracy of tissue sample analysis through advanced image processing techniques. Whole Slide Imaging (WSI) converts traditional microscope slides into high-resolution digital formats, enabling detailed examinations. This paper presents a brief experimental survey of instance segmentation models on two prominent histopathology datasets: PanNuke and NuCLS. Unlike previous surveys that merely describe deep learning models for general pathology images, we conduct experiments using state-of-the-art models including Mask R-CNN, Detectron2, YOLOv8, YOLOv9, and HoverNet on both datasets. Our study evaluates these models for both binary and multiclass instance segmentation tasks. The NuCLS dataset, featuring over 220,000 annotated nuclei from breast cancer histopathology images, is used for multiclass segmentation across 13 distinct nuclear classes. The PanNuke dataset, comprising 205,343 labeled nuclei across 19 tissue types, is employed for both multiclass and binary instance segmentation of five cell types: neoplastic, inflammatory, soft tissue, dead, and epithelial. We assess each model's performance using metrics such as mean average precision (mAP), F1 score, and Dice coefficient, providing a comprehensive evaluation of their strengths and limitations. The results of our study offer valuable insights into the capabilities of different instance segmentation models in histopathology image analysis. We observe varying performance across tissue types and cell categories, highlighting the importance of model selection based on specific histopathology tasks. Our findings aim to guide researchers in choosing appropriate models for their specific needs, ultimately contributing to the advancement of digital pathology and improving diagnostic accuracy in clinical practice. Also provides a foundation for future research in instance segmentation for histopathology images.