STFormer: Spatio‐temporal former for hand–object interaction recognition from egocentric RGB video

Abstract In recent years, video‐based hand–object interaction has received widespread attention from researchers. However, due to the complexity and occlusion of hand movements, hand–object interaction recognition based on RGB videos remains a highly challenging task. Here, an end‐to‐end spatio‐temporal former (STFormer) network for understanding hand behaviour in interactions is proposed. The network consists of three modules: FlexiViT feature extraction, hand–object pose estimator, and interaction action classifier. The FlexiViT is used to extract multi‐scale features from each image frame. The hand–object pose estimator is designed to predict 3D hand pose keypoints and object labels for each frame. The interaction action classifier is used to predict the interaction action categories for the entire video. The experimental results demonstrate that our approach achieves competitive recognition accuracies of 94.96% and 88.84% on two datasets, namely first‐person hand action (FPHA) and 2 Hands and Objects (H2O).