LaCLIP-FER: Landmark-Aware Adapting Vision-Language Models for Facial Expression Recognition

Facial emotion recognition (FER) is a crucial task for intelligent human-computer interaction, with the primary challenge being the development of discriminative facial emotion representations. However, the performance of FER is often hindered by inter-class similarity and intra-class variability, as individuals across different races, genders, and ages express emotions in diverse ways. To address these challenges, we introduce LaCLIP-FER, a method that leverages the powerful transfer capabilities of vision-language models (i.e., CLIP) while incorporating landmark information through parameter-efficient fine-tuning (PEFT). Specifically, we adapt CLIP for the FER task by designing a landmark-aware adapter (La-Adapter), which integrates invariant facial landmarks (e.g., eyes and mouth) to enhance the extraction of fine-grained facial representations without adding significant training parameters. Additionally, we introduce positive and negative prompt signals to improve the model's ability to distinguish emotional semantics and mitigate inter-class ambiguity. Experimental results on the RAF-DB and Affect-7/8 datasets demonstrate that LaCLIP-FER outperforms state-of-the-art methods, achieving WAR scores of 91.88%, 67.29%, and 63.12%, respectively.