IR50-LongCLIP: Enhancing CLIP's Facial Expression Recognition Capabilities through Long-Text Processing and IR50 Integration

We propose an innovative image-text matching model IR50-LongCLIP based on the CLIP framework. The model aims to improve facial expression recognition (FER) capabilities and proposes a new paradigm for facial expression recognition systems. The model framework consists of two main parts: the vision part and the text part. The vision part uses pre-trained IR50 on facial expression recognition to capture the multi-scale features of facial expressions. In the text part, we introduce a text-based facial expression description that presents the five facial senses—eyes, nose, mouth, and cheeks—that are closely related to the expression in the form of a long text description of the expression. This approach, unlike methods that rely only on emotion category name labels or truncated text descriptions, can more accurately convey the relationship between the five facial features and the expression of emotions in different expressions. In addition, we introduce a learnable prompting module to help the model understand the relationship between different categories of facial expressions during training. Extensive experiments confirm the effectiveness of the proposed method, and IR50-LongCLIP performs well in the RAF-DB benchmark test.