MediaPipe-based keyframe extraction method for hand grasping motions

To address the performance bottleneck of Vision-Language Models (VLMs) in robotic arm imitation learning caused by context length constraints, this paper proposes a heuristic keyframe extraction method based on hand motion features. The method utilizes the MediaPipe hand keypoint detector to acquire hand motion trajectories, constructs a temporal hand velocity curve, and employs Gaussian filtering with local extremum detection strategies to identify velocity trough frames highly correlated with operations such as grasping and placing. To enhance robustness, a dual-threshold filtering mechanism is introduced, incorporating spatial confidence and temporal interval constraints to screen candidate keyframes. Experimental results demonstrate that the proposed method achieves a precision of 0.87 and an F1-Score of 0.88, significantly outperforming traditional uniform sampling approaches. This data-driven adaptive keyframe extraction strategy effectively preserves critical operational semantics while reducing redundant frames, providing a reliable visual context for subsequent VLM-based task reasoning and instruction generation.