Facial Action Unit Recognition Guided by Labeling Rules

Existing facial action unit (AU) recognition studies either do not leverage the relations between AU representations or do not utilize important facial cues for AU labeling fully, which has limited their performance. To address these limitations, we design a novel AU recognition framework guided by AU labeling rules. Specifically, we first leverage AU labeling rules from the Facial Action Coding System to separate facial judgment areas and define the explicit correspondence between AUs and the judgment areas. A region feature extraction component is utilized to extract representations for the judgment areas. Then, AU-specific representations are mapped from their corresponding judgment areas. AU relations are further encoded to enhance the AU representation learning based on Transformer encoder. After that, we introduce a region relation learning component to encode the relations among judgment areas to further guide the region representation learning through Transformer encoder and the proposed auxiliary task. Finally, the encoded AU and region patterns are jointly fed into the AU predicting component to perform AU recognition based on Transformer decoder. The designed Transformer encoder-decoder framework can fully leverage both relations among AU representations and facial cues. Experimental results on three public databases demonstrate the effectiveness of the proposed method compared with that of current state-of-the-art methods.