Co2SAM: Exploring Co-Occurrence Challenges With SAM in Weakly Supervised Semantic Segmentation

Weakly supervised semantic segmentation builds a semantic segmentation model with only image-level annotations, which provide only categorical information without localization details. This limitation leads to persistent challenges related to class co-occurrence. Traditional methods often address this issue by incorporating external data or employing data augmentation techniques. In contrast, we benefit from the large and diverse amount of data seen during segment anything model (SAM) pretraining to exploit its potential recognition ability and solve the class co-occurrence problem with only the pretrained model. Furthermore, existing SAM-based methods typically adopt SAM to produce pseudo-labels, which are then employed to train separate segmentation networks without employing SAM as a backbone for direct segmentation with image-level labels. In addition, we found that although the SAM-based approach can solve the class co-occurrence problem, it still suffers from full target leakage. Based on these observations, we propose a one-stage, dual-teachers–one-student network architecture, termed Co2SAM. Specifically, we utilize Focal loss, Contrast loss, Dice loss, and Template loss, and fine-tune the image encoder with LoRA. Experimental results substantiate the effectiveness of our Co2SAM. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/chunmengliu666/Co2SAM</uri>.