Intelligent note recognition: enhancing vocal performance with convolutional neural networks

In this paper, we propose MT-CAT: a Multitask CNN–Attention Transformer network for simultaneously executing pitch classification, technique classification and aesthetic scoring in vocal performance recognition. An important and surprising result is that multitask learning substantially improves the performance for all tasks, even for the notoriously difficult aesthetic regression task, with an R2 value of 0.774 which is by far the best reported value on VocalSet. MT-CAT combines spectral–temporal feature extraction via convolutional neural networks (CNNs), long-range temporal modeling via Transformer attention and uncertainty-based dynamic loss weighting to automatically harmonize task learning. On the VocalSet dataset, MT-CAT achieves a macro-F1 score of 87.64% for technique classification and a top-3 accuracy of 84.0% for pitch classification, both results are superior to those previously reported state-of-the-art methods. Early stopping at epoch 35 successfully prevented overfitting and the model generalized well. These results illustrate that joint multitask architectures are able to model technical and perceptual dimensions of singing performance effectively, which could have highly beneficial applications in music pedagogy, voice therapy and computer-based singing evaluation.