Multitrack Music Transcription Based on Joint Learning of Onset and Frame Streams

Multitrack music transcription is the task of converting music recordings into symbolic music representations that are assigned to individual instruments. This task requires simultaneous transcription of note onset and offset events for individual instruments. In addition, the limited resources of many transcription datasets make multitrack music transcription challenging. Thus, even state-of-the-art transcription systems are inadequate for applications requiring high accuracy. In this paper, we propose a framework to jointly transcribe onsets and frames for multiple instruments by integrating a deep learning architecture based on U-Net with an architecture based on Perceiver, which is a variant of the Transformer architecture. The proposed framework effectively detects the pitches of different instruments by employing the multi-layer combined frequency and periodicity (ML-CFP) with multilayered frequency-domain and quefrency-domain features as the input data representation. Our experiments demonstrate that the proposed multitrack music transcription system outperforms existing systems on five transcription datasets, including low-resource datasets. Furthermore, we evaluate the proposed system in terms of instrument type and show that the system provides high-quality transcription results for the predominant instruments.