Few-Shot Image Classification Based on Multimodal Information Fusion

To address the underutilization of label text information in few-shot image classification, this paper proposes a multimodal information fusion method that effectively improves classification accuracy. Traditional few-shot classification approaches primarily focus on image features, often neglecting the rich information contained in labeled text. To remedy this, we introduce the CLIP model to map images and text into a unified feature space, aligning visual and textual data at the same semantic level. Based on this alignment, we construct multimodal category prototypes that integrate image and text features, thereby enhancing intra-class similarity and inter-class differentiation. Additionally, we design a masking enhancement module to strengthen the model’s feature extraction capabilities under incomplete information by randomly masking parts of the image and performing a recovery task. Experimental results show that our method outperforms existing approaches across multiple few-shot classification datasets, validating the effectiveness of fusing image and text information and offering new directions for few-shot classification research.