Accurate Hand Modeling in Whole-Body Mesh Reconstruction Using Joint-Level Features and Kinematic-Aware Topology

Whole-body mesh reconstruction utilizes neural networks to reconstruct the 3D human body, face, and hands, forming a fundamental task in computer vision. It is used to model human action in many practical applications that prioritize upper body action, particularly the hands. However, accurately estimating the 3D mesh parameters of hands in practical applications remains challenging due to severe self-occlusion and high self-similarity in hand action. To address these challenges, the Accurate Hand Modeling in Whole-Body Mesh Reconstruction (AHM-WBMR) is proposed in this article. It mainly consists of two innovative components: the joint-level features progressive matching and refinement and the kinematic features propagation. In the joint-level features progressive matching and refinement, the 3D deformable cross attention and the 3D deformable transformer decoder are proposed to assist in refining hand joint-level features. Further, in the kinematic features propagation, the kinematic-aware topology network is proposed to distinguish and relate different hand joint-level features using three types of kinematic topology structures. We evaluated AHM-WBMR on the UBody and FreiHAND datasets, both of which contain rich hand movements. Compared with the state-of-the-art methods, we achieved improvements of at least 10.2% and 11.1% in hand-related metrics on the two datasets, respectively.