RAFFNet: Restricted Attention Feature Fusion Network for Self-Supervised Image Representation Learning

Learning image representations with deep self-supervised models is an important task in computer vision, which aims to establish beneficial and general representations from unlabeled images. However, existing efforts train models mainly on high-level features, neglecting lower-level features and their global spatial information, thus limiting the discriminative power of the learned representations. In this work, we propose a representational learning model based on restricted attention feature fusion network (RAFFNet) to improve the quality and generalization of the learned image representations. Specifically, to fully exploit the features in the deep network, we use a self-supervised model on multi-level features to learn more general representations. Meanwhile, a new feature fusion strategy with a dual attention mechanism of channel and space is used for multi-level features, enabling the model training to obtain more important and comprehensive feature information. Furthermore, in order to better extract global spatial information, we devise a simple but effective attentional weighted mask, which restricts the weight of spatial attention and prevents the model from focusing only on local features with high attention weights. Experiments on four public classification datasets, CIFAR-10, CIFAR-100, Tiny ImageNet and ImageNet-1%, and two object detection datasets, PASCAL VOC and COCO, demonstrate that the proposed RAFFNet has better representation performance and generalization ability than most state-of-the-art image representation learning algorithms.