Late Fusion of Text and Images for Predicting User Interests in Pinterest Social Network

Due to the large amount of information on social media, predicting user interest becomes crucial for various tasks including the improvement of the user experience in these platforms. This paper introduces a late fusion model designed to predict one of 29 predefined categories (considered as the user interests) established by Pinterest for any new pin uploaded by a user. A pin is an information unit with two components: an image and a short text comment. Given a new pin, the proposed model performs a multimodal classification by transforming and classifying each data modality (image and text) independently and then fusing the category probabilities with a weighted sum to predict the final categories for the pin. To test the validity of the fusion, a set of experiments is conducted analyzing and comparing the performance of different state-of-the-art deep learning transformations for each modality; and using a logistic regression model for classification. Additionally, the fusion is evaluated under different values for a weighting factor, in charge of assigning less or more importance to each data modality in the final decision. The experiments are carried out over a dataset composed of 29,000 pins collected directly from the Pinterest website. The results show that the late fusion of image and text representations increases the accuracy of predicting user interests when compared to the use of each single modality.