A transformer-based intelligent online virtual conductor assistance system for choirs

The development of virtual technology and artificial intelligence has made the automation and intelligence of choral conducting an important direction for enhancing the expressiveness of online choral performances. Existing methods, however, often fail to generate conducting gestures that are both natural and closely synchronized with music. To this end, this study proposes a Conformer-based Virtual Conductor Motion Generation Model (CV-CMGM) based on the Convolution-Enhanced Transformer (Conformer), which is used to automatically generate a sequence of 3D conductor motions from music signals. The CV-CMGM model consists of a music and motion encoder, a cross-modal generator, and a local temporal and global content discriminator. The Conformer structure captures local and global temporal dependencies in music and motion sequences, enabling cross-modal generation from music to conductor motions. Experimental results on the ConductorMotion100 dataset show that CV-CMGM achieves a motion Fréchet Inception Distance (FID) of 27.96, a geometric FID of 19.35, motion diversity and geometric diversity of 5.94 and 5.12, respectively, and a beat alignment score of 0.214. CV-CMGM outperforms the comparative models in all indicators. User studies further indicate that the virtual conductor motions generated by CV-CMGM achieve scores above 4.4 in interpretability, dynamic expressiveness, musical structure coherence, and naturalness/smoothness, outperforming other models. These data indicate that the proposed method can effectively generate virtual conducting motions that match the musical content and exhibit good expressiveness.