Diverse image generation with diffusion models and cross class label learning for polyp classification

Pathologic diagnosis is a critical phase in deciding the optimal treatment procedure for dealing with colorectal cancer (CRC). Colonic polyps, precursors to CRC, can pathologically be classified into two major types: adenomatous (malignant potential) and hyperplastic (benign). Various imaging techniques, such as narrow band imaging (NBI) and white light imaging (WLI), are adopted in capturing polyp-specific features for accurate classification and have different advantages. However, the existing classification techniques mainly rely on a single imaging modality and show limited performance due to data scarcity. Recently, generative artificial intelligence has been gaining prominence in overcoming such issues, especially with various generation-controlling mechanisms using text prompts and images. However, such mechanisms require class labels to make the model respond efficiently to the provided control input. In the colonoscopy domain, such controlling mechanisms are rarely explored; specifically, the text prompt is a completely uninvestigated area. Moreover, the unavailability of expensive class-wise labels for diverse sets of images limits such explorations. This raises the key question of how diverse and clinically meaningful colonoscopy images can be generated in a text-controlled manner from limited annotated data. Therefore, in this work, we develop a novel model, PathoPolyp-Diff, that generates text-controlled synthetic images with diverse characteristics in terms of pathology, imaging modalities, and quality, enabling more effective augmentation of downstream diagnostic models. The proposed model follows a two-stage process: first, the model learns to distinguish polyp from non-polyp characteristics, and then it focuses on pathology-specific features. In the process, we introduce cross-class label learning to make the model learn features from other classes, reducing the burdensome task of data annotation. We validate the effectiveness of text-controlled synthesis and cross-class label learning by performing polyp classification (adenomatous/hyperplastic) with different imaging modalities (NBI/WLI) and text prompts. The experimental results show that incorporating the proposed synthetic images for data augmentation yields an improvement of up to 7.91% in balanced accuracy on a publicly available dataset, highlighting the utility of our approach for enhancing downstream classification performance. Moreover, cross-class label learning achieves a statistically significant improvement of up to 18.33% in balanced accuracy during video-level analysis. The code is available at https://github.com/Vanshali/PathoPolyp-Diff.