Estimation of Hand Pressure and Pose From RGB Images Based on Cross-Modal Cues

The hand is one of the most commonly used mediums for human-object interaction, and the sensing of hand force is an important research direction. Force measurement is traditionally realized by sensors, which is costly. Therefore, the visual-based method, which infers the hand pressure from RGB images, can be an effective way to estimate the contact force. To obtain a more accurate estimation of the hand tactile information through vision, the interaction of cross-modal information is designed into the model, and the shape and gesture information needed is introduced for the hand pressure measurement. The experimental results show that the new model designed outperforms the state-of-the-art (SOTA) model by 0.825%–5.93% in all important metrics on a large-scale dataset. In addition, the task of estimating hand posture and pressure from the same RGB image, as well as estimating the position of hand joints using regression models, can be obtained simultaneously.