A Comprehensive Approach for Real-Time 3D Model Manipulation Using Hand Gestures

This paper introduces a pioneering gesture-controlled interaction system for real-time 3D model manipulation, seamlessly integrating advanced computer vision, real-time communication, and web-based 3D rendering technologies. Leveraging MediaPipe and OpenCV, the system accurately captures and processes hand movements, translating them into predefined gestures such as fists or open palms. These gestures are transmitted via WebSocket, ensuring low-latency data transfer to a Three.js-based frontend, where they dynamically control 3D models through operations like rotation, scaling, and translation. The system distinguishes itself by combining lightweight, web-based technologies with robust gesture recognition, achieving 92.3% accuracy and 18.4 ms latency, outperforming similar systems reliant on proprietary hardware. The system’s modular architecture comprises four layers: gesture detection for hand tracking, communication for data transmission, processing for model manipulation, and presentation for an intuitive user interface. This framework delivers a touch-free, natural interaction experience, enhancing human-computer interaction (HCI) across diverse applications, including virtual learning, gaming, and assistive technologies. By enabling hands-on exploration of 3D models, it fosters immersive education and improves accessibility for users with physical disabilities. Despite its strengths, challenges such as gesture recognition accuracy under varying lighting conditions and hardware dependency persist. Future enhancements include integrating voice commands using VOSK to address recognition limitations, optimizing performance for complex models, and extending compatibility with augmented and virtual reality (AR/VR) to enhance immersion. This system sets a transformative foundation for intuitive, accessible, and immersive touch-free interaction, paving the way for next-generation HCI innovations.