A Multimodal Virtual Reality System for Switchgear Operation Training: Integration of Dynamic Gesture and Speech Recognition

This paper introduces a multimodal interaction system for switchgear operation training, utilizing virtual reality (VR) technology combined with gesture recognition and speech recognition techniques. The system provides an immersive training environment designed to improve operational accuracy and user engagement. Dynamic gesture recognition and two speech recognition models—K2 (Next-Gen Kaldi) and GMM-HMM with LSTM—were integrated to facilitate intuitive interactions. Experimental evaluations demonstrated that the system offers significant improvements in task accuracy, user experience, and training efficiency compared to traditional methods. The K2 model outperformed the GMM-HMM with LSTM approach in terms of speech recognition accuracy. Additionally, user feedback confirmed the effectiveness of the multimodal interaction system in simplifying complex operational tasks. Future work will explore the application of large language models to enhance natural language understanding and further refine the system's usability. This study highlights the potential of integrating advanced multimodal technologies into industrial training systems to enhance both safety and proficiency.