A new method for 3D face reconstruction using transformers based on action unit features

Abstract We present a novel 3D face reconstruction framework called Facial action unit (AU) feature-based 3D FAce Reconstruction using Transformer (AUFART) that can generate a 3D face model that is responsive to AU activation given a single monocular 2D image to capture expressions. We propose a novel 3D face reconstruction framework, called AUFART (Facial Action Unit Feature-based 3D Face Reconstruction using Transformer), which generates 3D face models responsive to AU activations from a single monocular 2D image, effectively capturing facial expressions. AUFART leverages AU-specific features as well as facial global features to achieve accurate 3D reconstruction of facial expressions using transformers. We also introduce a loss function designed to guide the learning process so that the discrepancy in AU activations between the input and rendered reconstruction is minimized. The proposed framework achieves an average F1 score of 0.39, outperforming state-of-the-art methods.