Learning transferable image-level features for zero-shot semantic segmentation

The current semantic segmentation neural networks have limitations in recognizing unseen classes. To address this problem, the recent studies draw attentions on zero-shot semantic segmentation. The general zero-shot learning works usually focus on image recognition, which requires to extract transferable features across classes from images. However, zero-shot semantic segmentation needs to transfer knowledge at the pixel level. As semantic classes are defined for the whole objects, it is intuitive that the image-level features are more transferable across classes than the pixel-level features. In this work, we propose the Class2Seg approach for zero-shot semantic segmentation based on the above intuition. The core idea of Class2Seg is to learn transferable image-level features to guide zero-shot semantic segmentation. Our approach contains two branches. One is the image-level classification branch, and the other is the semantic segmentation branch. A cross-task correlation layer is designed to fuse the transferable image-level features into the semantic segmentation branch to promote information transfer from source classes to target classes at the pixel level. Extensive experiments on the Pascal-VOC dataset clearly support the effectiveness of our approach.