Research on Multimodal Commodity Recommendation System Incorporating µ2Net+ (ViT-L/16)

随着各领域推荐系统的迅速发展，多模态推荐已经成为个性化推荐的下一个竞争赛道。在以往的神经网络推荐模型中总是采用单一的推荐模型，例如Bert4Rec，DLRM等，它们都存在一些限制，a) 单一架构无法透彻解析用户输入文本信息的意图，b) 总是将输入限制为单一的文本或者图像。为了解决这些限制，实现推荐系统从单模态向多模态的跨越，本文提出了KGBM4Rec模型，这是一种基于知识图谱、图神经网络、Bert序列推荐和µ2Net+ (ViT-L/16)的混合神经网络模型。该模型通过构建领域知识图谱，提高模型扩展性，利用图神经网络从知识图谱中提取用户及商品之间的关联特征；采用Bert序列模型对文本特征进行建模；µ2Net+ (ViT-L/16)利用具有一系列卷积层和池化层的U形网络架构从图像中提取层次特征进行建模。多模态的个性化推荐可以让用户体验得到极大提升，本文在淘宝直播多模态视频商品检索数据集(Taobao)、来自在线京东购物网站的Products-10K数据集上进行实验，最终实验结果表明，所提出的推荐模型在多个数据集上均取得了平均高于基线模型10%的推荐效果。With the rapid development of recommendation systems in various fields, multimodal recommen-dation has become the next competitive track for personalized recommendation. In the past, neural network recommendation models always used a single recommendation model, such as Bert4Rec, DLRM, etc. They all have some limitations: a) a single architecture cannot thoroughly analyze the intention of the user’s input text information, and b) the input is always limited to a single text or image. In order to solve these limitations and realize the transition of the recommendation system from single modality to multimodality, this paper proposes the KGBM4Rec model, which is a hybrid neural network model based on knowledge graph, graph neural network, Bert sequence recommendation and µ2Net+ (ViT-L/16). This model improves the scalability of the model by constructing a domain knowledge graph, and uses graph neural network to extract the correlation features between users and products from the knowledge graph; the Bert sequence model is used to model text features; µ2Net+ (ViT-L/16) uses a U-shaped network architecture with a series of convolutional layers and pooling layers to extract hierarchical features from images for modeling. Multimodal personalized recommendations can greatly improve user experience. This paper conducts experiments on Taobao live multimodal video product retrieval dataset (Taobao) and Products-10K dataset from JD.com online shopping website. The final experimental results show that the proposed recommendation model has achieved an average recommendation effect of 10% higher than the baseline model on multiple datasets.