MoE-based multimodal alignment for vocal metaverse classrooms

Immersive metaverse classrooms introduce new challenges for vocal music education, where effective instruction depends on the real-time coordination of breathing, facial expression, and speech. Existing virtual teaching systems typically rely on single-modal or static evaluations and fail to capture the fine-grained temporal coupling required for professional vocal training, resulting in fragmented feedback and limited pedagogical effectiveness. To address this gap, this paper proposes a real-time multimodal intelligent agent for vocal metaverse classrooms based on a sparse Mixture-of-Experts (MoE) architecture. The system integrates respiratory signals, facial expression dynamics, and speech semantics through heterogeneous encoders and dynamically routes multimodal representations to specialized teaching experts. This design enables low-latency multimodal alignment while supporting personalized instructional strategies aligned with core vocal pedagogy. Experiments conducted on a large-scale multimodal dataset involving 1,024 students demonstrate that the proposed method achieves a trimodal alignment score of 0.87, reduces breath–speech alignment error to 42 ms, and reaches 91.5% accuracy in expression–language alignment, while maintaining an end-to-end latency of 76 ms. Both quantitative evaluations and teaching assessments indicate that the system significantly improves feedback precision, instructional responsiveness, and learning effectiveness in immersive vocal training environments. These results suggest that integrating sparse multimodal intelligence with pedagogically driven expert specialization provides a practical and scalable solution for real-time, personalized instruction in vocal metaverse classrooms.