M-IRRA: A multilingual model for Text-based Person Search

Text-based person search (TBPS) is a crucial task in computer vision and natural language processing, with various applications such as security surveillance, social media analysis, and multimedia database management. Thanks to advancement in deep learning, TBPS has made significant progress in handling English queries. However, existing methods often struggle with multilingual queries due to the diverse linguistic structures, vocabularies, and cultural nuances. To overcome this challenge, in this paper, we introduce a multilingual model for text-based person search, named M-IRRA, built upon the IRRA (Cross-Modal Implicit Relation Reasoning and Aligning) model - proposed for text-based person search in English [1]. In the M-IRRA model, the sentence transformer Multilingual CLIP is utilized to extract the feature embedding from text. Then, a dense layer is added to match and align the features extracted from text and image. The proposed model can support queries in multiple languages, ensuring strong performance across various linguistic contexts. Extensive experiments are performed on 3000VnPersonSearch dataset for TBPS in Vietnamese and its derived version in English to show the effectiveness of the proposed framework on both monolingual and multilingual search scenarios. The results obtained for monolingual search in Vietnamese are 38.60%, 67.4% and 77.62% at at rank 1, 5 and 10 respectively, showing a significant improvement over the existing models.