Vision Transformer Based Hash Coding for Efficient Image and Audio Retrieval with Global and Local Equilibrium Distance Constraints

With the application of efficient retrieval in information systems and retrieval augmented generation with vector database for large language models, hash coding algorithms have made progress in recent years. The rise of transformer technology in the field of deep learning has brought the possibility to further improve the effect of hash coding algorithms. We introduce the vision transformer framework to both images and audios, and propose a novel approach for the tasks of multi-label image retrieval and audio event retrieval. In the proposed hash coding model, global and local equilibrium distance constraints are integrated, so that the hash codes for images can be better obtained through the global hash centers and local similar samples. In order to realize end-to-end training and hash code generation for audios, we adopt the adapter of mel spectrogram, thus the proposed approach can be simply converted and applied to audio hash coding. Comparative experiments verify that better results can be achieved on multiple image and audio datasets.