SUMMR: A Unified Multimodal Representation Framework for Songs

The understanding and representation of songs is a crucial issue in music platforms, as it can facilitate numerous applications in the music field. Songs are a common multimodal art form within the music domain, achieving rich musical connotations and strong expressiveness. However, the data of songs exhibit obvious multimodal and heterogeneous characteristics, presenting significant challenges to the understanding and representation of songs. Regrettably, the current methods do not respond to these challenges effectively. To this end, in this study, a unified multimodal representation framework for songs, namely SUMMR, is proposed. Specifically, first, a two-layer framework is put forward. In embedding layer, the features of different modalities data are embedded into a unified space. In content layer, a novel cross-modal attention mechanism is designed, which effectively capture the cross-modal semantic associations and deep music features, thereby obtaining a unified representation of songs. Then, a two-level hierarchical pre-training algorithm is proposed, which can effectively lower the training cost. Finally, experiments are conducted on two typical music tasks with public datasets of songs, where the experimental results demonstrate the effectiveness of SUMMR for understanding and representation of songs, and also show that SUMMR has good capability of being fine-tuned in many song-based tasks.