Information supplement based on Tip-adapter

The Contrast Language-Image Pre-training (CLIP) model learns to associate image and text content by pre-training on a large number of image and text pairs. The CLIP model can understand the relationship between image content and text description, thereby achieving functions such as image recognition, classification, and retrieval in various tasks. And due to its zero sample learning ability, it can perform effective task inference even without a large amount of labeled data. The proposal of CLIP model has brought new research ideas and application possibilities to the field of multimodal learning, and has shown excellent performance in the zero-shot domain. When we want to apply CLIP to the field of few-shot remote sensing image classification, the main challenge is how to fine-tune the knowledge stored in the clip so that the entire model can better adapt to downstream tasks and how to solve the problem of overfitting in training models under the situation of few shot samples. The commonly used CLIP-based fine-tuning methods have poor performance under few-shot conditions. In this paper, we propose a semi-supervised learning based cached model method, which classifies the test set for the first time based on a small amount of labeled data in the training set as the cached model. The high confidence pseudo labeled data in the test set is used as a supplement to jointly classify the remaining low confidence test samples. The experiment shows that our proposed innovative method has significantly improved accuracy compared to previous methods on two benchmark datasets.