Combining Handcrafted Spatio-Temporal and Deep Spatial Features for Effective Human Action Recognition

Abstract Human action recognition research has become increasingly sophisticated in recent years because it is applied in many different applications like surveillance, sports analysis, robotics, etc. Many approaches have been put up to solve the problem of human action recognition, but a number of challenges, such as occlusions, background clutter, environmental variations, camera motion, and intra- and inter-class similarities, still need to be resolved. Effectively collecting and combining the spatial–temporal information is essential for describing a video in action recognition. In this research paper, we address the problem of human action recognition by combining handcrafted spatio-temporal features with deep spatial features. This paper proposes a novel method for recognizing human actions in video by combining handcrafted spatio-temporal texture features extracted by our proposed feature descriptor, Volume Local Derivative Gradient Ternary Patterns (VLDGTP), and deep spatial features extracted from a modified Inception-v4 network. To reduce the dimension and to get equal sized feature vectors, we employed the PCA dimensionality reduction technique for both types of features. Then the dimensionality-reduced feature vectors are combined and passed to the SVM classifier for action recognition. Extensive experimentation is carried out on three benchmark datasets: KTH, UCF-101, and HMDB-51 datasets. Our proposed HAR method (VLDGTP + DEEP_FEATURES) outperforms existing HAR methods on the KTH, UCF-101, and HMDB-51 datasets, achieving an accuracy of 98.33% for KTH, 97.10% for UCF-101, and 87.50% for HMDB-51 dataset, demonstrating its superior performance in action recognition tasks.