A Reproducibility Study of Multimodal Embeddings for Recommender Systems

Abstract Adopting multimodal embeddings in recommender systems has become standard practice, typically leveraging text and visual embeddings derived from pre-trained multimodal foundation models. While these embeddings are widely assumed to improve recommendation performance, systematic empirical evidence remains limited. In this work, we present a comprehensive reproducibility study that critically examines the actual contribution of multimodal embeddings to recommender systems. Through large-scale experiments on 14 widely used state-of-the-art models, we evaluate the effectiveness of multimodal embeddings both jointly and per modality. Using a modality knockout strategy, an architecture-preserving diagnostic intervention, we measure model sensitivity to text and image embeddings by replacing them with constants or noise, rather than removing or altering architectural components. Our results indicate that multimodal embeddings generally improve performance, particularly in graph-based fusion models, while simpler fusion methods yield marginal gains that are difficult to attribute purely to multimodal semantics. Notably, text embeddings alone often match full multimodal performance, whereas image embeddings alone provide limited benefit. These findings clarify the practical value of multimodal embeddings and offer reproducible insights for future multimodal recommendation research. Our code and datasets are available under https://github.com/GAIR-Lab/MKF4MMRec .