AUD:AU-based Diffusion model for Facial Expression Synthesis from a Single Image

Facial expression editing technology can significantly enhance the human-computer interaction experience. Recent developments in Generative Adversarial Networks (GANs) have yielded remarkable outcomes in the realm of facial expression synthesis, by shaping the generation process with images featuring different emotions. Although successful, GAN-based methods are limited to producing a finite number of expressions based on the dataset’s contents. To overcome this constraint, this study introduces the AUD: AU-based Diffusion model for Facial Expression Synthesis. This novel approach comprises two main stages: modulation and editing. The modulation phase entails training through the re-targeting of a pre-existing diffusion-based language-image model for expression refinement. We employed an AU-aware fine-tuning strategy to enhance the distinction between AUs related to facial expressions and irrelevant characteristics. In the editing stage, an initial inversion technique is applied to convert the input image into latent noise. Subsequently, a set of text cues encompassing both the initial and target AU vectors are utilized for text-guided localized emotion enhancement, facilitated by attention control. Quantitative and qualitative evaluations indicate that our methodology, which is the first work to integrate diffusion models with expression editing, excels in generating high-quality images with subtle, continuous expressions, and emotion accuracy score(EmoACC) for specific generated emotions surpass the SOTA performance.