A Depth-Based Lightweight 3D Dynamic Hand Gesture Recognition Framework

Hand gesture recognition is a key technology in the field of human-robot interaction (HRI). This paper proposes a depth-based lightweight dynamic hand gesture recognition framework for HRI, which includes a 3D hand pose estimation network based on biogeometry constraints (BCPoseNet) and a dynamic hand gesture recognition network based on multi-stream residual attention (MRAGesNet). BCPoseNet leverages its lightweight multi-flow hierarchy structure to extract and refine the skeletons of palm and five fingers respectively. Meanwhile, a set of biogeometry loss functions are designed to further constrain the relative position of skeletons. MRAGesNet constructs a lightweight 1D ConvNet baseline, on which an adaptive cross-channel residual block (ACR) and multi-channel attention mechanism (MAM) are introduced to explore the linkage mechanisms of gesture semantics and further enhance recognition performance. A feature calculation component is designed which enables a seamless integration of the framework through the transformation of 4 functions. The proposed methods achieve the state-of-the-art performance in accuracy and speed that are evaluated extensively on 4 public datasets and 1 custom dataset. In addition, a hand gesture imitation experiment on a hand-arm robot platform proves the application performance of the proposed framework for HRI.