Holism To Atomism: Enhancing The Vision-Language Alignment For Cross-Modal Few-Shot Learning

Few-shot learning has made substantial progress due to the pre-trained cross-modal models, e.g., CLIP, which contrastively align the vision-language pairs. Simply aligning the class token of the image with the associated text suffers from the challenge, that the intrinsic mismatch between the vision and language features in a given pair is ignored. In this study, we propose a Holism-to-Atomism approach, called H2A, to address this mismatch challenge. During training, it first aligns the vision patches with the language tokens by determining the similarity matrix between patches and tokens. Meanwhile, the refined vision patches are scratched according to the similarity matrix, in order to represent the given training image compactly while removing the redundant patches. Then, the alignment between those refined patches and the other image patches is implemented, by calculating the patch-based prediction probability based on the image patch similarity. During inference, the queried images are classified according to the patch-based prediction probability combined with the existing class token-based one. Experiments on 11 datasets show that the H2A significantly improves upon the baseline approaches especially on 8-shot and 16-shot.