DMMSA net: a dual multi-modal self-attention network for lung target areas segmentation using chest radiology images and reports

Precise segmentation of lung target areas is crucial for the clinical diagnosis and treatment planning of pulmonary diseases. However, current methods mainly focus on using CNNs or Transformers for segmenting single-modal images, or on fine-tuning multi-modal pre-trained models for specific segmentation tasks. They either overlook the role of text in enhancing image representation, fail to provide a comprehensive representation of images from multiple dimensions, or neglect to capture the associations between key terms in the text and critical regions in the image, which is vital for improving model interpretability. To address these issues, this study proposes a novel framework, the Dual Multi-modal Self-Attention Network (DMMSA Net), aimed at improving the accuracy of lung target regions segmentation by integrating chest radiology images with accompanying reports. The DMMSA Net utilizes Deeplab-ResNet101 as a backbone to extract image features and combines them with word embeddings generated from radiological reports. A feature scaling/calibration module is employed to map both modalities into a unified feature space, followed by a dual multi-modal self-attention mechanism applied at both spatial and channel dimensions of image features. An adaptive gated fusion mechanism is then used to selectively merge features at different semantic levels for final segmentation. The proposed model shows significant improvements in lung target areas segmentation on public dataset Qata-Cov19 (Dice: 81.25%, mIoU: 71.79%) and proprietary clinical dataset GWCMC-PCR (Dice: 93.27%, mIoU: 87.76%). The DMMSA Net effectively enhances interpretability and generalizability in clinical practice by capturing long-range dependencies between image pixels and text descriptions through dual multi-modal self-attention.