Multi-Task Learning Approaches for Music Similarity Representation Learning Based on Individual Instrument Sounds

Music similarity representation learning (MSRL) based on individual instrument sounds (InMSRL) is a potential technique to develop a new function in music recommendation and retrieval systems, allowing users to focus on multiple partial elements of music pieces. There have been proposed two main approaches, Cascade that sequentially performs music source separation (MSS) and music similarity feature extraction for each instrument sound and Direct that directly extracts disentangled music similarity features. Each approach has a specific problem; in Cascade, a separation error often causes adverse effects on the feature extraction; and in Direct, it is usually hard to learn accurately disentangled features. In this paper, we propose multi-task learning approaches to leveraging MSS to address these issues. For Cascade, we propose end-to-end fine-tuning of the MSS model and the music similarity feature extractors using an auxiliary separation loss. For Direct, we propose multi-task learning based on the disentangled music similarity feature extraction and MSS based on reconstruction with the disentangled music similarity features. We conduct experimental evaluations and demonstrate that 1) the end-to-end fine-tuning for Cascade significantly improves InMSRL performance, 2) the multi-task learning for Direct is also helpful to improve disentanglement performance in the feature extraction, and 3) Cascade with the end-to-end fine-tuning outperforms Direct with the multi-task learning.