Gesture-to-Robot Mapping in a Mixed Reality System for Safe Bronchoscopic Teleoperation

In the field of surgical robotics, achieving intuitive, straightforward, and practical navigation and control remains a key research challenge. Natural Orifice Transluminal Endoscopic robotics, such as bronchoscopic robotics, typically rely on hardware interfaces like joysticks, with navigation based on endoscopic images displayed on a monitor. This mode of operation lacks intuitiveness and places significant demands on the surgeon’s expertise. To address these limitations, this paper presents a mixed reality (MR)-based control system for bronchoscopic robotics. The system introduces an intuitive master-slave control method based on key points extracted from hand gestures. To improve the robustness and safety of gesture-based control, a Kalman filter is employed in combination with a dynamic weighting mechanism that adjusts based on the deviation from the bronchial centerline. To further enhance spatial awareness and interaction experience, the system integrates the virtual bronchial tree and electromagnetic navigation sensors into HoloLens2 (a kind of head-mounted display). This integration enables remote, contactless control of the robot within an immersive MR environment. Experimental results validate the reliability of the proposed gesture-based teleoperation method and demonstrate the effectiveness of the integrated MR control system.