SemAlign: hierarchical semantic alignment for few-shot image recognition

Recognizing novel categories with limited training samples even a single instance per class remains few-shot learning (FSL) a challenging problem. Recent studies have explored leveraging semantic information such as word embeddings of class names to enhance model’s comprehension for obtaining more stable class prototypes. However, these approaches primarily utilize semantic information at high-level representation while neglecting the interaction between underlying knowledge. They remain constrained by pseudo-visual features learned from sparse labeled samples, leading to minimal benefits. In this study, we introduce a novel hierarchical semantic alignment (SemAlign) approach for FSL. Diverging from conventional methods that simply employ semantic knowledge to compensate class prototypes directly, we investigate aligning semantic features with visual features across different hierarchical levels to strengthen the representation of class-specific visual features. Specifically, we design two complementary semantic alignment mechanisms: 1) global-level semantic alignment through generating semantic feature attention to refine the visual feature of the image and 2) patch-level semantic alignment that aligns semantic information with localized visual features of image patches. By integrating these two mechanisms, the visual prototypes are enhanced and more generalizable. Comprehensive experiments conducted on two benchmark datasets highlight the exceptional performance of our approach, achieving average accuracy improvements over most existing methods.