How Image Segmentation Affects Classification: Implications in a Cancer Diagnostic Model

This study investigates how segmentation impacts the performance of Vision Transformer (ViT) models in classifying histopathological images. We trained ViT on segmented images using the DigestPath dataset with expert-annotated masks and the BACH dataset with generated masks and then tested the model on unsegmented images. We compared the accuracy of this pipeline with another pipeline where the training was conducted on unsegmented images. Our results showed that segmentation affected classification differently depending on the dataset. In DigestPath, training on segmented images led to lower accuracy (62.12%) compared to unsegmented training (83.33%), likely due to overfitting on highly accurate ground-truth masks. Conversely, in BACH, training on segmented images improved accuracy (81.25% vs. 76.25%), suggesting that less accurate masks could enhance generalizability. These findings highlight the nuanced impact of image segmentation on the performance of classification models, particularly in histopathological image analysis.