Prior-guided attention fusion transformer for multi-lesion segmentation of diabetic retinopathy

To solve the issue of diagnosis accuracy of diabetic retinopathy (DR) and reduce the workload of ophthalmologists, in this paper we propose a prior-guided attention fusion Transformer for multi-lesion segmentation of DR. An attention fusion module is proposed to improve the key generator to integrate self-attention and cross-attention and reduce the introduction of noise. The self-attention focuses on lesions themselves, capturing the correlation of lesions at a global scale, while the cross-attention, using pre-trained vessel masks as prior knowledge, utilizes the correlation between lesions and vessels to reduce the ambiguity of lesion detection caused by complex fundus structures. A shift block is introduced to expand association areas between lesions and vessels further and to enhance the sensitivity of the model to small-scale structures. To dynamically adjust the model's perception of features at different scales, we propose the scale-adaptive attention to adaptively learn fusion weights of feature maps at different scales in the decoder, capturing features and details more effectively. The experimental results on two public datasets (DDR and IDRiD) demonstrate that our model outperforms other state-of-the-art models for multi-lesion segmentation.