Zero-Shot Visual Classification Through Discriminative Text Descriptor Selection

Vision-language models such as CLIP have demonstrated remarkable zero-shot classification performance by aligning visual and textual representations. Recent studies have shown that enriching class descriptions with large language model-generated visual descriptors can further improve classification accuracy without additional training. However, existing descriptor-based methods use all pre-generated descriptors equally for every test image, regardless of which descriptors are actually useful for a given input. This is problematic in fine-grained classification, where visually similar classes share many common attributes, causing non-discriminative descriptors to reduce the influence of truly informative ones. The purpose of this study is to address this issue by proposing CoDES (confusion-aware discriminative descriptor selection), a method that dynamically selects discriminative descriptors at test time. For each test image, CoDES first identifies confusing classes and then selects descriptors that are effective in resolving the confusion. Experiments demonstrate that CoDES consistently outperforms existing methods across eight benchmark datasets, achieving competitive performance against descriptor selection methods requiring labeled data without using any labeled data. These results indicate that the proposed CoDES can effectively improve zero-shot classification performance by selecting descriptors that are helpful for distinguishing between confusing classes.