Multimodal Deep Content‐Based Music Recommendation System Using MIDI and Lyrics

ABSTRACT Nowadays, recommendation systems are used in various web applications, which include online music streaming services. Suggesting relevant music in a web application is an open research problem with numerous promising results reported. The most common approach for recommendation systems is learning user preferences, which are based on various modalities. There has been more focus on user modalities like rating information and online reviews to understand the user's music preferences. The proposed model uses multimodal auxiliary song information (MIDI and lyrics) to develop the feature representations, improve user satisfaction with the generated recommendation, and overcome the recommendation system's cold‐start problem. Our novel contribution to this work is our multimodal music recommendation system, which captures musical features using MIDI data and semantic features using lyrics in an attempt to provide music recommendations with different multimodal fusion techniques. We also present a comparative analysis of varying word embedding models for musical lyrics to analyze which model performed best for our multimodal music recommendation system. The proposed model is a hybrid two‐stage music recommendation model that adequately leverages multimodal item embedding representation to improve recommendation performance substantially.