Learning multi-scale attention network for fine-grained visual classification

Fine-grained visual classification (FGVC) is a very challenging task due to distinguishing subcategories under the same super-category. Recent works mainly localize discriminative image regions and capture subtle inter-class differences by utilizing attention-based methods. However, at the same layer, most attention-based works only consider large-scale attention blocks with the same size as feature maps. And, they ignore small-scale attention blocks that are smaller than feature maps. To distinguish subcategories, it is important to exploit small local regions. In this work, a novel multi-scale attention network (MSANet) is proposed to capture large and small regions at the same layer in fine-grained visual classification. Specifically, a novel multi-scale attention layer (MSAL) is proposed, which generates multiple groups in each feature maps to capture different-scale discriminative regions. The groups based on large-scale regions can exploit global features and the groups based on the small-scale regions can extract local subtle features. Then, a simple feature fusion strategy is utilized to fully integrate global features and local subtle features to mine information that are more conducive to FGVC. Comprehensive experiments in Caltech-UCSD Birds-200- 2011 (CUB), FGVC-Aircraft (AIR) and Stanford Cars (Cars) datasets show that our method achieves the competitive performances, which demonstrate its effectiveness. This is the overall framework of MSANet. The multi-scale attention module, including the multi-scale attention layer and the feature fusion stage, can be added into deep convolutional neural networks. We utilize ResNet-50 as an example. Given an image, we utilize ResNet-50 to obtain feature maps of the last convolutional layer. Then, multi-scale attention layer is utilized to capture different scales discriminative features. And, the feature fusion is used to fuse different scales discriminative features. The Conv Block consists of 1 × 1 convolution to decrease the channels of the concatenated feature map, a batchnorm and a dropout. Finally, the fused feature is fed into a classifier to get the final class prediction. • A novel multi-scale attention module is proposed, which contains a multi-scale attention layer based on the interlaced sparse self-attention scheme. The module can accurately localize discriminative parts to improve the performance of fine-grained visual classification. • A novel multi-scale attention layer is proposed to generate multi-scale features that can capture global information and subtle local information. This allows networks to focus on features at different scales at the same layer.