RGB image-based hand rotation angle estimation for teleoperation

Purpose This study aims to achieve precise hand rotation angle regression using an economical RGB camera and apply this technology for the teleoperation of Elite robot in tasks requiring posture adjustment. Design/methodology/approach Leveraging the benefits of affordable RGB cameras while preserving the natural movement of human hands in teleoperation, the authors propose a novel regression model. It features an innovative hand distance metric and a dual-stream convolutional architecture for extracting global visual features. These are integrated using a weighted fusion structure to enhance angle regression accuracy. Findings Through ablation studies and teleoperation application experiments, the author demonstrated that distance metrics based on the middle finger, coupled with ResNet-18 within our dual-stream model, significantly improve performance. Incorporating a weighted fusion structure achieved a minimal Mean Absolute Error of 1.53° and a maximum accuracy of 75.00% within a 2° error threshold. Furthermore, this model is suitable for teleoperating Elite in precision tasks like spooning and pouring sugar, demonstrating the effectiveness of the proposed method. Originality/value This paper presents an advanced model for hand rotation angle regression using an RGB camera, combining global visual and local geometric features. Through a weighted fusion strategy, regression precision is significantly improved. To the best of the authors’ knowledge, this is the first instance of such high-accuracy regression achieved with a low-cost camera, offering a viable and innovative solution for teleoperation systems with limited budgets that also need to maintain human-like habits, which lays a data foundation for the application of Large Models in robotics.