AUMEs: AU Detection-Based Dual-Stream Multi-task 3DCNN for Micro-expression Recognition

Micro-expressions are brief, involuntary facial movements that can reveal real emotions. However, their short duration and low intensity pose a challenge for feature extraction and learning of neural networks. To overcome this challenge, we propose AUMEs, a 3DCNN-based multi-task learning framework that utilizes deep learning-based Lagrangian motion magnification and optical flow computation methods to enhance spatio-temporal features of micro-expressions, thus solving the problem of weak micro-expression motion intensity. AUMEs also use AU detection as a parallel task to improve the accuracy of micro-expression recognition by transferring knowledge from the AU detection task, and focal loss is utilized in model training to handle category imbalance in the micro-expression dataset. AUMEs achieve competitive results compared with existing SOTA methods on the CASMEII and SAMM datasets, achieving accuracy (Acc.) of 81.05% and 79.85%, UF1 score reaches 0.8880 and 0.7450 on the five-category task, and on the three-category UAR reached 89.02% and 75.86% and 0.8880 and 0.7450 for UF1. Furthermore, in both dataset analyses, the multi-task approach surpassed the single-task method across both the five-category and three-category classifications.