KF-TMIAF: An Efficient Key-Frames-Based Temporal Modeling of Images and AU Features structure for Masked Facial Expression Recognition

Masked Facial Expression refers to facial expression that people deliberately make which do not correspond to their true emotions. The purpose of masked facial expression is to conceal one’s genuine feelings. Precisely identifying masked facial expression aids in revealing the genuine emotions, and finds applications in diagnosis of mental illness. However, the masked facial expression sequences have redundant information, temporal modeling is difficult, and the expressive ability of AU features needs to be improved. To solve the existing problems, we first propose a Key frame Selection and Data Augmentation (KS-DA) method to obtain the key frame in the sequence. In addition, we propose a Global Feature Relation Block (GFR) to aggregate global information and combine it with Temporal Convolutional Networks (TCN) to better realize temporal modeling. Finally, we propose an Adaptive Weight Generation Module (Ada-WGM) to form different weights from adaptation, allowing for better extraction of the temporal changes in AU features. Finally, the experiments have shown that the proposed method has improved the mixed expression classification task (36 task) by 15.23% and the experienced emotions classification task (6E task) by 17.77%, respectively, and obtain sort results.