NEHand: Enhancing Hand Pose Estimation in the Wild through Synthetic and Motion Capture Datasets

Recovering interacting hand poses in natural environments poses a highly challenging task. Current datasets for interacting hands are relatively simple, lacking complex textures or background interference, allowing models to focus more on hand interaction behaviors. However, the diversity of backgrounds and textures in natural environments can significantly interfere with hand pose estimation, affecting the model’s generalization ability. Therefore, we propose an approach named NEHand, which combines synthetic datasets with publicly available datasets, including those based on motion capture, to achieve the recovery of interacting hand gestures in natural environments. To simplify the model, reduce complexity, and enhance generalization, we employ a strategy of flipping the left hand to the right hand to effectively achieve the recovery of both hands. Additionally, we introduce constraints from the mano model, including adjustments to joint angles, finger bending, and overall hand shape. This ensures that the generated hand poses not only conform to anatomical structures but also represent common, spontaneously occurring postures in our daily lives. By restricting and adjusting these parameters within specified ranges, we consider the natural motion relationships between hand joints, ensuring that the generated hand poses are physiologically reasonable. The method undergoes quantitative and qualitative evaluations, showcasing state-of-the-art robustness, generalization, and accuracy across multiple diverse datasets. This indicates that our approach has achieved significant success in addressing the challenge of recovering interacting hand poses in natural environments.