RGB-Guided Spatial-Spectral Fusion for Microscopic Hyperspectral Image Segmentation

Tumor segmentation in histopathological images is crucial for clinical cancer diagnosis and treatment. Digital imaging of pathology sections can simultaneously capture microscopic hyperspectral images (MHSIs) and RGB images. MHSIs, with their combined spatial and spectral information, are popular in many emerging methods. However, the corresponding RGB images, which contain richer spatial morphological information such as textures and colors, are often overlooked. In this study, we propose using RGB data as an auxiliary modality to enhance tumor segmentation by providing detailed spatial features that complement the spatial-spectral features of MHSIs. To facilitate effective inter-modal feature fusion between MHSIs and RGB images and bridge the modality differences between 3D and 2D data, we design a spatial-guided alignment module (SGAM). It derives spatial guidance weights and enhanced details from RGB images, which are fused with MHSIs to obtain discriminative spatial-spectral features for tumor identification. Additionally, we develop a hybrid loss function to balance pixel predictions between tumour and background regions, and encourage the model to focus on boundary areas. Experiments on a public histopathology dataset show that our approach can identify tumors with high accuracy and outperform several state-of-the-art (SOTA) methods using one modality.