Lightweight multi-scale convolution with blended feature attention for facial expression recognition in the wild

Abstract Facial expression recognition (FER) has achieved excellent performance in recent years under the controlled scenarios through deep learning methods. However, the accurate recognition of facial expression in the wild conditions with occlusion, pose changes, and uneven lighting still a challenging problem, not to mention the problem of limited computing resources faced by the growing size of proposed network models. To solve these problems, this paper proposes a multi-scale network based on lightweight convolution (MLC-Net), aiming to improve the recognition accuracy of FER in real-world environments while significantly reducing the number of parameters. In MLC-Net, image shallow features are extracted for global and local blocks through pre-extracted blocks. The global feature extraction block uses a mixed washing network as the basis of the multi-scale module, reducing the its parameters and computational complexity when extracting different levels of semantic information. Meanwhile, the improved efficient lightweight channel-spatial attention module (SFAM) is used to enhance the feature fusion ability of the multi-scale module. The local feature extraction block utilizes convolutional groups and lightweight SFAM to extract and enhance local features, guiding the network to pay attention to regions with significant features, and proposes a local relationship transformer, through which a multi-head attention mechanism is used to establish connections between regions, thus further enhancing the ability to recognize complex expressions. The effectiveness of the proposed MLC-Net is validated on multiple in the wild FER datasets, and the results show that MLC-Net can achieve a good balance between recognition accuracy and network lightweighting, providing a promised solution for practical application of FER.