Multi-Task Hybrid Conv-Transformer With Emotional Localized Ambiguity Exploration for Facial Pain Assessment

Recently, there has been significant progress in automatic pain assessment based on facial expression analysis. However, the performance of pain assessment remains unsatisfactory, due to a lack of analysis on local pain-related action units and emotional ambiguity. In particular, ambiguous pain expressions complicate the estimation of pain. It is argued that certain facial local regions related to pain should receive more attention while estimating pain intensities. Based on this, we propose a multi-task hybrid Conv-Transformer method for facial pain assessment, which utilizes the self-attention mechanism to explore facial local features related to pain intensities and constructs a multi-task joint optimizing module to mitigate facial emotional ambiguity. In particular, the proposed method modifies the network structure of the vision transformer model to better estimate continuous pain intensities. Meanwhile, a multi-task module is constructed to jointly optimize the classification and the regression tasks of pain assessment, which effectively regularizes the extracted features and facilitates a better fit of the regressed prediction to the given label. Finally, experimental results on the UNBC Pain dataset illustrate that the proposed method performs better with pain assessment compared with state-of-the-art methods.