Automated music genre classification using multimodal mel-spectrogram representation learning with transformer-enhanced convolutional recurrent networks

Music Genre classification is a significant problem in the field of music information retrieval (MIR) and artificial intelligence, especially in very large digital music collections. Traditional methods are typically based on manually created descriptors and manual analysis which may be less scalable and less able to capture the complex spectro-temporal patterns. This study presents the multimodal deep learning system for automatic music genre classification called TransCRNN, based on the combination of transformer and convolutional-recurrent neural networks. The proposed framework gradually embeds CNN-based local spectro-temporal feature extraction, BiLSTM-based bidirectional temporal modelling, attention-based feature weighting and Transformer-based long-range contextual learning. The output of the deep representation is combined with 57 handcrafted acoustic features, to fully use complementary information in Mel-spectrogram and acoustic representations. Experiments were run on the GTZAN dataset with 10 genres of music and 70%/15%/15% training, validation and test split. Across five repeated experimental runs, TransCRNN achieved a mean accuracy with 93.33% accuracy, 94.69% precision, 93.33% recall, and 94.00% F1-score. Further ablation, modality and robustness analysis and computational analyses were performed to investigate the contribution and stability of the framework proposed. Moreover, SHAP and LIME were used to explain the model predictions on the global and local levels. The results are interpreted within the scope of GTZAN only, about the generalization of results across datasets being an important direction in future research.