MAGAE: Multi-Level Alignment Over Aggregation Semantic Graph With Attribute Enhancement for Text-Based Vehicle Retrieval

As cities and traffic become increasingly digitized, vehicle retrieval technology has emerged as a vital component of Intelligent Transportation Systems (ITS), aiming to identify target vehicle tracks or clips from calibrated road cameras. Recently, the task of text-based vehicle retrieval has garnered increasing attention, which is more applicable compared to image-based vehicle retrieval, i.e., vehicle re-identification. However, the text-based vehicle retrieval is also facing great challenges due to the huge semantic gap between images and texts. In this paper, we propose a novel method, called multi-level alignment over aggregation semantic graph with attribute enhancement (MAGAE), for text-based vehicle retrieval task. Specifically, we employ large language models (LLMs) and pretrained visual models to extract global vision and text feature embeddings, respectively. Then, an F-Encoder architecture is developed to dynamically integrate local and global features of vehicles. Furthermore, we design a co-attention learning mechanism to extract and enhance the robustness of vehicle attribute embeddings. Additionally, a multi-level semantic graph alignment module is introduced to effectively bridge the significant gap between vision and text through attribute aggregation and interaction. Finally, to improve the learning capacity of the network, we employ the merits of multi-task learning by incorporating four learning loss functions. Experimental results on the CityFlow-NL benchmarks validate the effectiveness of the proposed model. Findings of this study may assist urban traffic management, particular in providing guidance in capturing hit-and-run vehicles.