Dual-Stream Networks for Human Activity Recognition and Interaction with Surena-V Humanoid Robot

Human activity recognition (HAR) plays a crucial role in advancing humanoid robot interaction, providing an intuitive interface for human-robot collaboration. This study proposes a dual-stream neural network approach, combining video frames and optical flow to interact with the Surena-V humanoid robot through visual commands. A dataset comprising 692 video clips of five actions-Come, Go, Left, Right, and Turn-was collected from 15 participants. Four models were implemented: a frame-based model, a motion-based model relying on optical flow, a combined model integrating both spatial and temporal information through EfficientNet and a Bidirectional GRU, and a compiled model incorporating a text encoder based on the Universal Sentence Encoder (USE). The combined model significantly outperformed the others, achieving an accuracy of 87%. The study highlights the effectiveness of integrating spatial and temporal data for more robust and accurate HAR in real-time robotic collaboration systems.