Virtual staining of Endoplasmic Reticulum Fluorescence image from Brightfield images using Transformer based GAN

Abstract Endoplasmic Reticulum (ER) plays a critical role in cell function and drug response. Fluorescence based cell painting enables detailed visualization of ER morphology and cell response to external perturbations. However, acquiring the fluorescence images for large scale experiments is resource intense and time consuming. In this work, a transformer based Generative Adversarial Network that integrates the convolution operation with the attention mechanism is introduced to predict the fluorescence image from the brightfield images. The generator follows an encoder decoder architecture, where the encoder utilizes Swin transformer blocks, enabling the model to capture local and global features. Further decoder reconstructs the fluorescence image using transposed convolution layers. The discriminator utilizes a patch-based network that evaluates the structural similarity in the local regions between the predicted and ground truth images. Quantitative evaluation demonstrates the reconstruction accuracy and structural similarity by achieving PSNR of 33.64, SSIM of 0.92, MSE of 0.000573 and MAE of 0.0114. Further, visual comparison evaluates the morphological consistency in the generated image. Additionally, feature based analysis using Cell Profiler resulted in high correlation between predicted and ground truth images. This indicates the ability of the model to preserve the biological information. Thus, the proposed approach provides an effective and reliable tool to traditional fluorescence imaging, enabling label free prediction of fluorescence image from brightfield images.