TB-SMGAN: A GAN Based Hybrid Data Augmentation Framework on Chest X-ray Images and Reports

Data augmentation is a common practice in image classification, employing methods such as reflection, random cropping, re-scaling, and transformations to enhance training data. These techniques are prevalent when working with extended real-world datasets, focusing on improving classification accuracy through increased diversity. The use of Generative Adversarial Networks (GANs), known for their high representational power, enables learning the distribution of real data and generating samples with previously unseen discriminative features. However, intra-class imbalances in augmentations are problematic for conventional GAN augmentations. Hence, we propose a framework named Text-Based Style-Manipulated GAN augmentation framework (TB-SMGAN) aims to leverage the generative capabilities of StyleGAN2-ADA. In this framework, we utilize StyleCLIP to control disentangled feature manipulations and intra-class imbalances. We enhance the efficiency of StyleCLIP by fine-tuning CLIP with x-ray images and information extractions from corresponding medical reports. Our proposed framework demonstrates an improvement in terms of mean PR-AUC score when employing the text-based manipulated GAN augmentation technique compared to conventional GAN augmentation.