Multimodal Item Retrieval and Matching Algorithm Based on Visual-Semantic Joint Embedding

This paper introduces a multimodal item retrieval and matching framework based on a visual-semantic joint embedding architecture. The proposed model leverages a dual-encoder structure — comprising a ResNet-50 visual encoder and a DistilBERT text encoder — followed by modality-specific projection heads to map features into a shared embedding space. Training is driven by a symmetric contrastive loss enhanced with hard negative mining and semantic regularization, fostering strong intra-class cohesion and inter-class separability. Evaluations on MS-COCO and Flickr30K demonstrate the model’s superior retrieval performance. Specifically, it achieves Recall@1 scores of 80.5 (I2T) and 68.4 (T2I) on MS-COCO, and 69.8 (I2T) and 55.7 (T2I) on Flickr30K, outperforming baselines such as UNITER, CLIP, and SCAN. The model also shows high stability and efficiency, converging faster than alternatives and requiring less training time. Ablation studies confirm the effectiveness of contrastive loss, projection heads, and semantic regularization, while sensitivity analysis identifies an optimal temperature parameter ([Formula: see text]). Training loss and ranking distribution analyses further validate the model’s robustness and fine-grained alignment capabilities. Overall, the proposed method offers a scalable, interpretable, and efficient solution for cross-modal retrieval tasks, with strong potential for deployment in real-world applications involving fine-grained and domain-specific queries.