Online Classification of Human Gestures Through Event Camera Data Using a 3DCNN

In environments where humans and robots interact and collaborate, the robot needs awareness of human actions and intentions to guarantee operational effectiveness and agent safety. In this paper, an online human action recognition system from event camera data is presented. A 3D Convolutional Neural Network (3DCNN) is built to classify event-based videos of isolated primitive assembly actions (idle, pick, place, screw), and then leveraged into classifying videos of action sequences. The 3DCNN is integrated into a system capable of acquiring, processing, and classifying real-time event data of assembly tasks. The proposed online system classifies the streamed data every 200 ms without accumulating delay.