A Coarse-to-Fine Multi-Hypothesis Method for Ambiguous Hand Pose Estimation

In hand pose estimation, challenges such as occlusion often result in partial observation of a human hand, making it difficult to uniquely determine the hand pose, thus leading to ambiguity in certain hand regions. Heatmap-based methods may struggle with locating ambiguous joints and end up violating physiological constraints in their predictions. Parametric model based single-solution methods often fail to adequately address this ambiguity issue due to the inherent one-to-many mappings between input and output, resulting in unstable regression. While some existing multi-hypothesis methods have improved diversity by directly modeling the distribution of ambiguous hypotheses, their localization accuracy still falls short compared to the recent single-solution methods. To achieve quality results in both diversity and accuracy, we propose a novel multi-hypothesis approach for hand pose estimation, by progressively integrating heatmap information into the distribution of ambiguous poses using a RANSAC-like strategy. It starts with a conditional-flow model to provide an initial estimate of a coarse distribution over ambiguous joint poses. This is followed by randomly sampling multiple hypotheses, projecting each of them onto 2D heatmap plane, and employing consensus checks to identify unambiguous joints that adhere to skeletal constraints. Joint features are then resampled, with mismatches due to incorrect estimations being eliminated. Finally, we refine the distribution of ambiguous poses using graph neural networks and attention mechanisms. Extensive empirical experiments are carried out, where our approach are carefully examined both qualitatively and quantitatively. It is shown to not only produce more diverse & feasible pose hypotheses than existing multi-hypothesis methods, but also achieves accurate localization results comparable to the state-of-the-art single-solution methods.