ModalSync: Synchronizing User Behavior with Multimodal Features for Multimodal Pre-training Recommendation

In the continually evolving landscape of online platforms, integrating multimodal information into recommender systems offers a promising avenue to enhance our understanding of user preferences and product insights. Traditional models primarily rely on user-item interactions, but the advent of multimodal systems utilizes additional data modalities-text, images, audio, and video-to enhance recommendation accuracy. However, existing multimodal recommendation architectures often fail to fully exploit the potential synergy between multimodal feature extraction and recommendation processes, leading to domain bias and false positives. In this work, we introduce ModalSync, a novel multimodal pre-training framework designed to synchronize multimodal features with user behaviors, closely aligning with human perceptual processes. Unlike previous approaches that utilize pre-trained generic encoders, ModalSync incorporates a pre-training method that integrates both unsupervised and supervised strategies, thereby fostering a harmonious relationship between interaction graphs and multimodal data. Our framework uniquely addresses domain bias by infusing recommendation-specific interaction data into the feature extraction process and reduces false positives by directing encoder attention towards crucial attributes. Furthermore, ModalSync introduces a staged co-training module that strategically adjusts the training dynamics of the feature extractors and GNNs, promoting an effective and seamless fusion of multimodal information. Extensive experiments across three public datasets demonstrate that ModalSync significantly outperforms existing methods, achieving state-of-the-art results.