SSTar-TransGAN: Transformer-based Unsupervised Multi-modality Hyperspectral Digital Staining Network

Immunohistochemistry (IHC) plays an important role in accurate cancer screening and diagnosis, the clinical application of which has been restricted by complex operation process, high costs and demand of professional skills from pathologists. Digital staining methods based on deep learning provide the possibility for Hematoxylin & Eosin (H&E) stained images to be converted into IHC stained images, but it needs to train multiple staining networks for various modalities of IHC images from different cancers, thus weakening the versatility and convenience of digital staining in the process of practical application. At the same time, microscopic hyperspectral imaging technology can provide abundant spectral information for pathological images, which has been proved effective in digital staining tasks to transform staining modalities from one to many, but microscopic hyperspectral imaging has also been troubled by time-consuming acquisition process and huge data storage. In order to overcome the above challenges, we propose SSTar-TransGAN network for digital staining tasks. With the inhabitation of StarGAN structure, SSTar-TransGAN transfers the training burden of generators into lightweight style encoders between different modalities, while in addition to the introduction of transformer structure into the encoder, the application of spectral super-resolution Swin-Spectral Transformer U-Net (SSTU) network enables the convertion from H&E stained RGB images into hyperspectral images as well, which ensure the spatial structure and color information of IHC images under multiple staining modalities and the further conversion between different staining modalities. Qualitative and quantitative experiments prove the performance of SSTar-TransGAN on digital staining tasks superior to other state-of-the-art digital staining methods.