A Virtual Reality Gearbox Experiment Platform Based on Hand Tracking and Natural Language Processing

In manufacturing engineering education, the gearbox is an essential mechanical component and a fundamental skill students must master. Recent data suggests that insufficient understanding of its load capacity and exceeding operational limits can damage internal components, potentially leading to severe consequences like fires. In order to address the challenges faced by physical laboratories and existing traditional virtual labs in the new engineering disciplines, including the lack of insufficient experimental safety, operational real-ism, insufficient personalized learning support, and the difficulty in creating immersive learning environments, we introduce a Virtual Reality Gearbox Experiment Platform Based on Hand Tracking and Natural Language Processing (VRGEPHTNLP). This platform aims to enhance the realism of detailed interactions, provide personalized learning experiences, and create immersive environments, thereby improving students' learning efficiency and practical skills. In order to enhance the teaching of gearboxes, we have developed the VRGEPHTNLP by integrating hand-tracking technology, the GPT3.5 large language model (GPT3.5 LLM), and Azure technology. This application allows for comprehensive observation and assembly-disassembly teaching functionalities of parts. We offer an interaction experience rich in detail and environmental immersion, alongside personalized feed-back and guidance in response to student inquiries. Compared to traditional computer-based virtual labs and conventional VR laboratories, our VRGEP-HTNLP exhibits significant advantages in improving user interaction realism, enhancing personalized learning support, and optimizing the immersion of the learning environment.