PI-Net: Privileged Information Assisted Referring Image Segmentation Network

Referring Image Segmentation (RIS) aims to segment the corresponding region of a visual entity described by input referring expressions. Traditional approaches separately extract visual-linguistic knowledge from pre-trained models, neglecting the corresponding multi-modal information. In this paper, we propose a Privileged Information assisted RIS network (PI-Net) aimed at enhancing the text-to-pixel multi-modal alignment capability. The incorporation of privileged information within visual-linguistic pre-trained models not only enriches the intrinsic feature extraction process but also enhances generalization capabilities. Furthermore, we propose a Global Grouped Spatial and Channel Attention (GGSCA), which serves to accentuate vital features while concurrently curtailing the salience of irrelevant features. To demonstrate the superior adaptability of our model on complex scenarios, we introduce a proprietary dataset called Ref-Hard. PI-Net shows best performance on Ref-Hard while also achieves 74.14%, 65.27%, and 63.56% respectively on RefCOCO, RefCOCO+ and RefCOCOg.