Modality Fusion with Triplet Contrastive Learning and Cross-Attention for Multimodal Recommendation

Given the challenge of sparse user-item interactions, modern recommendation systems increasingly utilize the multimodal data (e.g., text and image) of items to enhance their performance. Features for modalities are first extracted independently using pre-trained models and subsequently integrated into graph-based models to learn node-level embeddings. The potential limitations of this method primarily manifest in its possible disruption of the semantic consistency of textual features and the integrity of visual features. Consequently, this may hinder recommendation systems from accurately retrieving items that match requirements for multimodal features based on node-level embeddings. To this end, this study proposes TriCAM, Triplet-based Cross-Attention Model with Multilayer Perceptron (MLP), consisting of a multimodal fusion module, an interest learning module and an interest matching module. The multimodal fusion module leverages an attention-based mechanism to align and integrate textual and visual modalities. The user interest learning module applies the triplet contrastive learning to make user representations closer to multimodal features of their preferred items. An individual interest is aligned with multimodal features of items by the interest matching module, which employs a MLP to capture high-order interactions among features. The experimental results on four publicly available datasets show the superiority of our method in terms of accuracy.