ClassifierAlign: Unsupervised multi-label image classification via training-free vision-language classifier alignment.
Existing vision-language models (VLMs) exhibit strong zero-shot generalization, yet remain fundamentally limited in Unsupervised Multi-label Image Classification (UMIC), primarily due to their inadequate multi-object perception and the intrinsic modality gap between vision and language. In particular, global representations tend to entangle multiple concepts, hindering accurate recognition of co-existing objects. To address these challenges, we revisit UMIC from a decomposition-then-classification perspective. Instead of predicting multiple concepts holistically, we first decompose an unlabeled multi-label image into multiple single-concept visual instances, thereby explicitly improving multi-object perception. We then perform classification based on aggregated predictions. Building upon this paradigm, we propose ClassifierAlign, a fully training-free framework that jointly leverages visual foundation models (VFMs) and VLMs. Specifically, we derive a lightweight visual classifier directly in the visual feature space, and semantically align it with CLIP's textual classifier, effectively bridging the vision-language modality gap without updating parameters of the pretrained foundation models. This design further enables seamless integration of powerful self-supervised VFMs (e.g., DINO), yielding robust and transferable representations without requiring spatial alignment. Extensive experiments demonstrate that ClassifierAlign achieves competitive performance with fine-tuning-based methods, while significantly reducing computational overhead and eliminating the need for costly annotated data. The source code for this work can be found at https://github.com/dengxili/ClassifierAlign.
Authors
- Chengfu yang (ORCID: https://orcid.org/0009-0002-9073-2688)
- Nanqing Liu (ORCID: https://orcid.org/0000-0001-7564-4896)
- Xili Deng
- Yaping Zhang
- Zhanxuan Hu
- Yi Zhao
Institutions
- Yunnan Normal University (CN)
Publication Details
- Journal
- PubMed
- Published
- 2026-11-14
- DOI
- https://doi.org/10.1016/j.neunet.2026.109901
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- article
- Field-Weighted Citation Impact
- 0.00