Heterogeneous Gap Bridging in Cross-Media Retrieval via Deep Association Learning with Joint Distribution and Semantic Alignments

Recent advancements in Internet technologies, particularly in social networks such as Weibo and Douban, have led to a substantial increase in multimodal data, including images, videos, audio, and text. This has resulted in growing research interest in multimedia information retrieval, with particular focus on cross-media retrieval. Central to this study is addressing the “heterogeneous gap” problem in cross-media retrieval. To address this challenge, we introduce a deep association learning method based on the nonlinear modeling capabilities of deep neural networks. Our approach integrates five multimedia types—images, videos, text, audio, and 3D models—for cross-retrieval. First, we employ a cross-media recurrent neural network (RNN) to capture fine-grained contextual information within different media types. Second, we propose a joint cross-media association loss function that combines distributional and semantic alignment mechanisms to learn cross-media associations both within and across media types. Third, we incorporate semantic category information into the association learning process to enhance the model’s semantic discrimination capability, thereby improving cross-media retrieval accuracy. Experimental results on the XMediaNet dataset demonstrate that the proposed method achieves higher mean average precision (mAP) values compared to existing cross-modal retrieval methods.