A study on a visualization teaching system for regional music based on the multimodal MusicFormer

To solve the problem of the intuitive identification of the stylistic features of regional music education, which is characterized by the fragmental expression of audio, sheet music, lyric semantics and regional cultural tags, this study proposes a regional music visualization teaching model based on the multimodal MusicFormer. To make regional music features seamlessly transform from the data ingestion to the music classroom, the model is based on the multi-head attention mechanism of the Transformer, additionally with the addition of audio spectrogram encoding, music score structure encoding, lyric semantic encoding, and regional label constraints. This created a technical framework which included a “modal-specific encoder—cross-modal attention fusion—space-constrained decoder—visualization mapping”. A dataset comprising 3,260 regional music samples, 18,450 audio segments, and 9,680 classroom interaction records was constructed.Experiments demonstrate that the F1-scores of the model for regional style recognition, music genre recognition, modal structure recognition, rhythmic pattern recognition and performance context recognition are between 89.98 and 94.63%, the modality alignment similarity of the fused representations is 0.891, which outperforms the modality alignment similarity of audio spectrogram features (0.873), modality alignment similarity of musical score structure features (0.841), and modality alignment similarity of lyric semantic features (0.806), and the visualization mapping error is reduced to 0.076. In the concurrent experiment with 126 participants, the average response time of the system’s interface was 226 ms, and the rate of task completion was 94.28%, which shows that this model has the ability to improve the performance in regional music feature recognition, visual expression and classroom interaction support.