PRSF: Pitched-Noise-Robust Score Following Based on Multimodal Learning

Score following is an important task in the field of music information retrieval, which maps the real-time audio input of a musical performance to the corresponding musical score. In the real practice of score following, due to imperfect audio capture or the difficulty of digital score acquisition, the input audio may contain other pitched sounds beyond the score, such as background accompaniment or the sounds of other musical instruments in an ensemble. However, existing methods are limited in resisting this kind of noise, which may lead to unstable score following for the target instrument. To address this issue, this paper proposes a pitched-noise-robust score following (PRSF) method based on a cyclic feedback and early fusion mechanism that cyclically connects traditional and deep models. First, a joint local pitch inference (JLPI) deep model is proposed to address the problem that traditional models have difficulty in discriminating the target sound source under pitched noise. This model is designed to infer the pitch of the target instrument based on multimodal learning, which is accompanied by a specially designed training strategy. Second, in order to sufficiently utilize the audio and score information and achieve robust score following, a hidden Markov model (HMM)-based score position inference model is introduced. For the first time, our PRSF method solves the score following task in orchestral performance scenarios where only the score of the target musical instrument is informed. Using cross-dataset evaluation, our proposed method achieves an align rate of 75.5% within a threshold of 300 ms under the pitched noise environment of orchestral music.