Improvement of Mediapipe Hands Network Structure

Considering the reduced positioning accuracy of MobileNetV2, the backbone network of Mediapipe Hands module, for small-scale joint points (such as fingertips and knuckles), especially in complex backgrounds or occluded scenes, the CA attention mechanism module is introduced. CA decomposes channel attention into horizontal and vertical position codes, which can more accurately capture the subtle displacement of finger joints. When detecting fingertips, the CA attention mechanism prioritizes the spatial gradient changes in the direction of finger extension, reducing computational complexity while preserving spatial information of direction sensitivity. In response to the weak recognition ability of Mediapipe for occluded joint points, this paper introduces the ERD model, LSTM, and position encoding, which can significantly improve the recognition ability of Mediapipe Hands module for occluded joint points. The ERD model quickly captures hand movement trends through efficient temporal modeling to predict the position of occluded joint points, the LSTM uses long short-term memory units to learn the motion rules of historical frames to infer the dynamic changes of occluded joint points, and the position encoding enhances the model's perception ability of joint point spatial distribution. Combined with temporal information, it can more accurately locate occluded areas. The synergistic effect of the three effectively solves the problem of joint point loss or false detection caused by occlusion, significantly improving detection robustness.