Abstract LB115: Unpaired image-to-image translation for unsupervised object segmentation in digital histopathology images

Abstract Introduction: AI tools for image segmentation require large training datasets that are annotated with many examples of objects of interest, e.g. manual annotated cell nuclei for training a model for nuclear segmentation. Such public datasets exist for H&E stains and well performing models for that stain are therefore easy to train and are also available as open source. For other stains, repeating the task of manual labeling a sufficient number of images is expensive and time consuming. Instead of training a new model for each new dataset, we propose a domain adaptation approach, where new datasets are translated to the domain of the dataset for which a good model is already available. This is achieved by training a model for unpaired image-to-image translation, that retains the biological content of the image, but translates the ‘style’ between domains. Methods: The Image-to-Image translation module consists of a deep neural network called CycleGAN (Cycle Consistent Generative Adversarial Network) aimed at translating an input image, from unlabelled dataset B, to another image that looks similar to images in the labeled dataset A. This allows for using the segmentation model previously trained on A to segment the translated input image. Importantly, the CycleGAN has to preserve the content of the input image (e.g., all objects and their layout should be preserved), since our goal is to eventually segment objects in this image. Instead, the image’s “style” (e.g., color, illumination) should be changed to look like images in the dataset A. In order to ensure that, we make several important modifications to the original CycleGAN model: normalization of feature maps to statistics of a virtual batch for generator networks, or dropping the feature map normalization completely in both generators and discriminators in favor of parameter normalization. We also add channel and spatial attention to the generator network as it is deeper and harder to train than the discriminator. We use bidirectional spectral normalization for parameter normalization. We also use UNet for the generator instead of the original ResNet. Results: Our system was trained to translate from IHC to H&E images StarDist was trained on and segmenting with StarDist on translated images. The results were evaluated on 300 IHC images that were manually labeled for the purpose of this work. Our results (precision: 0.89, recall: 0.77, F1 score: 0.82) are better than the best previously reported (CycleGAN + CellPose - precision: 0.63, recall: 0.49, F1 score: 0.55). Conclusion: Using our method, a rapid adaptation of a new image domain to the one for which good segmentation models already exist, allows for reliable reuse of the existing model without the need for any labeling of the new dataset. Citation Format: Nikola Andrejić, Milica Spasić, Igor Mihajlović, Petra Milosavljević, Uros Milivojević, Danilo Delibasic, Ivana Mikić, Siniša Todorović. Unpaired image-to-image translation for unsupervised object segmentation in digital histopathology images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 2 (Late-Breaking, Clinical Trial, and Invited Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_2):Abstract nr LB115.