Concept fine-tuning concern correction for composed cross-modal person re-identification

Composed cross-modal person re-identification aims to simultaneously localize and identify the target person based on composed text-image information. Language–image co-supervised person retrieval guides spatial localization of target description by learning similarity from image–text pairs. Nevertheless, the state of the art suffers from apparent semantic gaps between visual and textual modality: plenty of visual concepts appearing in images are missing in their paired text description, such as background and neighbors. Such overload semantics misalignment circulates in pre-training, leading to an inferior retrieval performance in dense predictions due to insufficient visual concepts captured from textual representations. Concept fine-tuning based on de-background information is proposed to close such a semantic gap. It is a pipeline that leverages de-background information as prompting to combat the residual local misalignment. For each image–text pair, a de-background concept archive is established to maintain potential visually matched concepts, leveraging the proposed text-driven fine-tuning and text-to-vision-guided ranking. In addition, a dataset called NCCMReid is created and expanded. Experimental evaluations of the proposed dataset demonstrate the effectiveness of the proposed method.