An Image Emotion Classification Framework Based on Instruction‐Guided Triplets With Descriptive Captions

ABSTRACT Image emotion classification remains a challenging task due to the intrinsic subjectivity of emotional perception and the semantic ambiguity inherent in visual content. Although recent studies have applied language supervision to exploit semantic cues, but most methods rely on fixed templates that exhibit limited emotional relevance, while generating high‐quality affective textual descriptions typically involves substantial manual effort and cost. To overcome these limitations, this paper proposes an emotion classification framework that integrates language supervision with instruction tuning. The proposed approach significantly improves emotion classification performance through three key components: (1) instruction‐guided generation of descriptive emotion captions (DECs), (2) cross‐modal pseudo‐label construction, and (3) adaptive multimodal fusion. We design emotion‐centric instructional prompts to guide large pre‐trained vision‐language models in generating semantically rich DECs, thereby surpassing the constraints of conventional template‐based methods. Instruction tuning is further employed to generate structured emotion pseudo‐labels, forming image‐caption‐pseudo‐label triplets that strengthen cross‐modal alignment. Finally, an adaptive fusion mechanism combined with a multi‐branch loss function is introduced to optimize classification efficacy. Extensive experiments conducted across multiple domain‐specific datasets demonstrate that our method achieves state‐of‐the‐art accuracy. Ablation studies confirm the critical role of multimodal collaboration in enhancing model performance. Furthermore, detailed linguistic analysis shows that DECs achieve high levels of emotional expressiveness, as evidenced by their length, degree of abstraction, and affective distribution, closely approximating the quality of human‐authored descriptions. This work demonstrates fine‐grained emotion description generation without manual annotation and offers a potential solution for multi‐source image emotion recognition.