Joint Attention Mechanism and Multi-task Learning for Weakly Supervised Skin Image Segmentation

In recent years, the application of deep learning technology in the field of medical image segmentation has become increasingly mature and has made certain progress. Applying this technique requires the use of a large number of medical image datasets with pixel-level annotations, however, the cost of pixel-level annotation of medical images is high. For this situation, we propose a weakly supervised medical image semantic segmentation model based on attention mechanism and multi-task learning. The attention mechanism can be used to focus on the information that is more critical to the current task in a large number of input information, and multi-task learning can share information among related tasks to promote network learning. As a result, the model can achieve better segmentation performance by using only image-level groundtruth labels. Since semantic segmentation task and saliency detection task are dense pixel prediction tasks,and class activation map (CAM) extracted from classification task is also used to represent pixel-level semantic information, it shows that these two tasks are highly related to the semantic segmentation task. Based on this premise,We propose the module of finding similarity from attention(SFA), which is used to calculate the similarity between multi-task feature maps to learn the correlation between tasks, and use similarity to optimize the prediction results of specific tasks. In addition, in order to extensively explore context relationships in dense pixel prediction tasks to achieve more accurate prediction results, we also propose a global context module (GCM) that pays attention to the global information of dense prediction tasks, captures long-distance dependencies in images, and improves the segmentation performance of the network. We did a lot of experiments, our method achieves 68.56% and 62.32% Miou on ISBI2016 and ISIC2017 datasets, respectively, and F1 -score is 78.25% on PH2 data set, significantly outperforming several recent state-of-the-art weakly supervised semantic segmentation methods.