Phantom Hand: Research on Dual-Arm Robotic Fine Teleoperation Based on VR Immersive Interaction and Adaptive Shared Control

Abstract Operations in chemical accident sites, nuclear facilities, and other hazardous environments may expose personnel to toxic substances, radiation, high temperatures, or unstable structures. Teleoperation offers a way to keep the operator at a safe distance, but controlling two robotic arms naturally remains difficult because multiple joints, grippers, and viewing directions must be coordinated at the same time. This study develops Phantom Hand, a dual-arm robotic teleoperation system that combines wearable inertial sensing with virtual reality interaction. Five IMUs worn on the chest, upper arms, and forearms capture the operator’s upper-body motion. The reconstructed arm movements are mapped to the two robotic arms, while a VR headset controls the remote camera and two handheld controllers operate the grippers. Several practical problems arise when human motion is transferred directly to the robot. The five IMUs share one RS-485 bus and are read sequentially, so their measurements are aligned using timestamps and short-term quaternion extrapolation. Natural-pose calibration and a chest-centered reference frame are used to reduce mounting errors and the influence of torso rotation. Swing–twist decomposition is then applied to separate coupled shoulder and forearm rotations. The control system also checks data validity, joint limits, command velocity, and communication status. Valid commands follow the operator’s motion, whereas unsafe or incomplete commands are constrained or held. Experiments showed that the average sampling-time span across the five IMUs was 112.43 ms. After calibration, spatial compensation, and temporal alignment were applied, the mean motion crosstalk decreased from 18.7% to 5.4%, while static jitter decreased from 0.684° to 0.241°. The robot also completed a grasping-and-transfer sequence under combined arm, gripper, and visual control. These results show that the proposed system can support natural and fine-grained dual-arm teleoperation using relatively low-cost sensing and interaction hardware.