HandRT: Simultaneous Hand Shape and Appearance Reconstruction With Pose Tracking From Monocular RGB-D Video

We propose a method to reconstruct a personalized hand avatar, representing the user's hand shape and appearance, from a monocular RGB-D video of a hand performing unknown hand poses under unknown illumination. Our method, HandRT, jointly optimizes hand pose, shape, appearance, and lighting parameters using a physically-based shading model in a differentiable rendering framework incorporating Monte Carlo path tracing. HandRT extends our previous work, Intrinsic Hand Avatar, by relaxing the assumption of a known coarse hand pose and utilizing depth data in the optimization. Specifically, we introduce an articulated registration energy based on iterative closest point over the depth point cloud that enables reconstruction from unknown hand poses via tracking across frames. Thus, we can reconstruct the avatar from arbitrary poses with high accuracy without relying on an off-the-shelf 2D joint detector at each frame. Further, HandRT is capable of precisely tracking the reconstructed avatar from either RGB or RGB-D input. Our evaluation demonstrates that our method outperforms existing hand avatar reconstruction methods on all commonly used metrics while producing significantly accurate mesh compared to state-of-the-art hand mesh recovery methods by a large margin on public and our captured datasets.