Spatio-Temporal Analysis in Video for Gesture Recognition using 3-Dimensional Convolutional Neural Network with Long Short-Term Memory

Nowadays, the gesture recognition in smart interfaces has gained significant attention for allowing the users to interact with devices in a more natural and spontaneous way. Also, the spatio-temporal analysis in video plays a crucial role in gesture recognition where it involves understanding the spatial and temporal relationships between body parts as well as movements. However, the existing Hidden Markov Model (HMM) faced challenges in detecting gestures where the variations in lighting and backgrounds affected the performance. Hence, this research proposes a combination of 3-Dimensional Convolutional Neural Network with Long Short-Term Memory (3D-CNN-LSTM) to model the temporal dependencies by resisting lighting and background variations. Initially, the input data is collected from 20BN-Jester dataset which consists of short video clips and fed into preprocessing where, data augmentation is applied to increase the data quantity. After that, the extraction of key features extracts the salient frames where the major motion occurs to reduce redundancy. Finally, the proposed 3D-CNN-LSTM is employed to learn spatio-temporal features by capturing information from frames and the fully connected layer with SoftMax activation function classifies the hand gestures. From the results, the proposed 3D-CNN-LSTM established better results when compared to existing 2D-CNN in terms of accuracy (98.9%) respectively.