Revisit Weakly Supervised Hashing With Deep Multi-Modal Foundation Models

Vision-Language Pretraining (VLP) has developed a series of fancy foundation models, which continuously advance the state-of-the-art on various multimodal tasks. However, there has been limited exploration of their potential for large-scale image retrieval. In a real-world image retrieval system, images are collected together with user-annotated tags from the web. These tags contain various information about the corresponding image and could be used as weak supervision for image representation learning. In this paper, we seek to harness the powerful image-and-text alignment ability of VLP foundation models to enhance compact image representation. Specifically, we propose a new weakly supervised hashing framework, which learns a deep hashing network and enhances weak supervision alternatively. First, we extract the image and tag representation from VLP foundation models, and learn the deep hashing network with a policy gradient process, which directly optimizes the retrieval performance, i.e., mAP. Then given the learned deep hashing network, we further enhance the weak supervision with a separate probabilistic decision process. This process also optimizes the retrieval performance by the ground-truth defined with the learned hashing network. These two processes are alternatively repeated until a fixed number of steps. Experiments on public image datasets prove the effectiveness of our method.