Channel Importance Adapter for Test-Time Adaptation in Vision-Language Models

Vision-language models (VLMs) have demonstrated impressive performance in zero-shot classification, where the model classifies images using only a list of class names. However, pre-trained models like CLIP often struggle when faced with distribution shifts in real-world applications. To address this challenge, we propose CIA, a training-free channel importance adapter, which dynamically adjusts the value of the feature channels of the image during the inference phase, enabling the model to rapidly adapt to changes in distribution in unseen domains. Unlike existing test-time adaptation methods, our approach requires no additional training or complex optimization, significantly reducing computational overhead. By applying feature channel transformation functions, the adapter quickly modifies image feature representations, enhancing the model’s classification performance across diverse scenarios. Extensive experiments have shown that our CIA is successful in adjusting image features, greatly enhancing the adaptation capacity of CLIP and achieving state-of-the-art performance in two challenging benchmarks with 15 common datasets.