Improving chinese cross-modal retrieval with multi-modal transportation data

Abstract As societal development progresses and individual travel needs evolve, the demand for acquiring multi-modal transportation status information has steadily increased. The present domain of transportation status encompasses a wealth of multi-modal information, including vehicle trajectory data, traffic condition visual imagery, and textual information. Acquiring multi-modal transportation status information facilitates a rapid understanding of the prevailing traffic conditions at a given location. In this study, we investigate multi-modal transportation status data encompassing trajectories, textual descriptions, and visual imagery. We establish a multi-modal dataset comprising circuit, text, and image data and enhance the state-of-the-art cross-modal retrieval model within the Chinese domain. We utilize hash codes to compute the similarity between images and textual descriptions. During model evaluation using our collected dataset, the Chinese-CLIP-Hash model (our model) demonstrates performance improvements over the Chinese-CLIP model. Specifically, it exhibits a 1.0% improvement in R@10 for the Text-to-Image task and a 0.4% improvement in R@5 for the Image-to-Text task, along with a 0.2% enhancement in R@10. Furthermore, during the cross-modal retrieval process, it achieves a notable speed increase, exhibiting an 11% improvement in speed compared to the Chinese-CLIP model when handling queries from a set of 400 textual descriptions.