MMFNet: A Multi-modal and Multi-attention Fusion Network for Multi-task Skin Disease Classification

Skin diseases rank among the most prevalent ailments in humans, which underscores the critical importance of early detection and diagnosis. Considering clinical practice, the integration of information from various modalities holds significant potential to enhance diagnostic accuracy and precision. In this study, we propose a multi-modal and multi-attention fusion network for multi-task skin disease classification. We initially devise the Shifted Window Cross-attention Fusion (SWCF) module based on the shifted window self-attention mechanism, leveraging the advantages of the attention mechanism to learn the correlations between different modalities and fully integrate multi-modal features. Subsequently, taking into account the heterogeneity of the data, we employ the Heterogeneous Data Cross-attention Fusion (HDCF) module using the idea of information decoupling and separation processing to integrate image and text data. Additionally, we suggest the Dynamic Loss Weight Allocation (DLWA) method for multi-task learning to refine the training procedure. We confirm the superiority of the proposed method on the publicly available multi-modal skin lesion dataset, Derm7pt. The average accuracy of multi-modal skin disease classification is 79.14%, surpassing the current state-of-the-art methods.