Open-Vocabulary Multi-Label Classification Based on Cross-Modal Feature Reconstruction Learning

Open-vocabulary multi-label classification aims to identify the labels for all significant objects of interest in the scene, including new objects unseen in the training set. Recent studies have demonstrated the potential of pre-trained Vision-Language Models (VLMs) in open vocabulary multi-label classification. However, their performance is often compromised because existing VLMs, such as the CLIP model, are not well-suited for capturing detailed local information. Multi-label image classification requires finding precise correspondences between class labels and local object regions, which can vary in shape and location. Therefore, it is a new challenge to learn this dynamic correspondence effectively. To address this issue, we develop a novel method called cross-modal feature reconstruction (CMFR) to determine which class label is present in the input image and simultaneously identify the local image regions associated with this class label. This dynamic association, represented by an association matrix, is learned from the following CMFR task: reconstructing the text feature of the class label from the set of transformer token features of the input image. Then, class labels with minor reconstruction errors are assigned to the input image. Our experimental results indicate that our method outperforms MKT methods in GZSL, improving the mean Average Precision (mAP) by 2.2% and 2.1% on the NUS-WIDE and Open Images datasets, respectively.