CLIP as Auxiliary Supervisor for Weakly Supervised Semantic Segmentation

Weakly supervised semantic segmentation (WSSS) with image-level labels effectively alleviates the time-consuming and laborious issue of manually annotating image pixels in semantic segmentation. However, the class activation map (CAM) only highlights the most discriminative regions, resulting in an incomplete pseudo-label, which provides insufficient supervision for training the semantic segmentation network. In recent years, the Contrastive Language Image Pre-Training (CLIP) model has received widespread attention due to its powerful zero-shot learning capability. Based on this, we propose a CLIP as auxiliary supervision framework (CAS) that utilizes CLIP as an auxiliary task to enhance supervision information, providing more accurate pseudo-labels for segmentation training. Specifically, we propose a Auxiliary Mask Generation (AMG) module, which uses CLIP to generate additional auxiliary masks and designs an auxiliary loss to supervise the generation of higher quality segmentation pseudo labels. Moreover, we propose a CAM Fusion Module (CFM), which leverages CLIP's potential for downstream tasks to enhance the localization ability of CAM. To further enhance the supervision provided by CLIP, we propose a Dynamic Mask Noise Filtering module (DNF), introducing Gaussian Mixture Modeling to dynamically reduce the noise in auxiliary masks. Large-scale experiments prove the effectiveness of the proposed CLIP as an auxiliary supervision framework, we validate the reliability of our method on the PASCAL VOC 2012 dataset and demonstrate significant improvements in segmentation performance.