Dual-branch skeleton–RGB fusion for human action recognition with spatio-temporal attention

Skeleton-based action recognition has gained attention due to the simplicity and robustness of skeletal data. However, the lack of fine-grained human information in skeleton data limits its performance. To overcome this limitation, We introduce a video modality and proposes a dual-branch network that models RGB video and skeleton data separately. For skeletal data, a graph convolutional network based on a self-attention mechanism is employed, achieving state-of-the-art performance in skeleton-based action recognition tasks. For video data, a 3D convolutional network is used for feature extraction. Additionally, we design a deep supervision method to oversee the intermediate features of both modalities, enhancing feature coupling and improving network efficiency. Experimental results demonstrate that the proposed dual-branch network shows significant advantages in jointly modeling video and skeleton data, achieving higher recognition accuracy and stronger generalization ability compared to traditional unimodal methods.