Polymorphic multi-head attention aggregation network for skin lesion segmentation

Skin cancer is one of the most common malignant tumors, and accurately segmenting the lesion area from dermoscopic images is of great clinical significance. With the rapid development of computer technology, Transformer-based models have dominated the field of automatic skin lesion segmentation. However, Transformer-based models typically focus on capturing global dependencies, lacking explicitly encoded convolutional layers as in RNNs or CNNs. This paper proposes a polymorphic multi-head attention aggregation network for skin lesion segmentation (PMAA-Net). It establishes a polymorphic multi-head attention mechanism (PMA), where each self-attention head is designed with convolutional layers that can encode gradients and textures to obtain various local features. Meanwhile, two attention heads are designed to capture spatial and channel-wise features respectively. This allows the model to capture local features and spatial-dimensional features. We further introduce a multi-head attention aggregation module (AGG) to aggregate multiple attention heads. This transforms the attention maps into a high-rank mixture distribution, significantly enhancing the feature representation capability of the attention heads. Our experimental results on three publicly available skin lesion segmentation datasets show that PMAA-Net outperforms other mainstream methods, especially those focusing on extracting local structural features. The codes are available at https://github.com/yuli0501/PMAA-Net.