Weakly supervised semantic segmentation via language-driven labeling and pseudo mask noise mitigation

Weakly supervised semantic segmentation (WSSS) using image-level labels has received widespread attention. Existing WSSS methods mainly train segmentation models based on pseudo masks generated by Class Activation Maps (CAMs). Most existing works are dedicated to addressing the incomplete activation in CAMs, but the generated CAMs often falsely activate non-target classes and background parts related to target objects, resulting in category confusion. In addition, the pseudo masks generated by CAMs often contain noise, which can also hinder the training of the final segmentation network. To alleviate the category confusion in CAMs, we utilize the potential of Contrastive Language-Image Pre-training (CLIP) and combine it with a large language model (LLM) to generate the most appropriate text label representation for each category, further improving the quality of the generated CAMs. In addition, to further mitigate the harm of noise in the generated pseudo masks, we introduce an adaptive mask noise filtering module and a dynamic weighted symmetric cross entropy loss over the segmentation network training process, which can further improve the quality of segmentation results by the loss feedback of the segmentation network. We validated the effectiveness of our method on the PASCAL VOC 2012 and MS COCO 2014 datasets, demonstrating impressive performance over existing WSSS methods.