EAGLE: Enhanced Activity Recognition in Occluded Environments With Improved GAIN and Bi-LSTM

Activity recognition (AR) using surveillance cameras remains challenging because of the variation between the data required for deep learning and the capabilities of CCTV-based frameworks for data recording. Most research on skeleton-based activity focuses on improving recognition using complete data, often overlooking performance on incomplete skeleton data due to occlusion or noise. However, in practical situations, it is inevitable to obtain a noisy or occluded human skeleton. When some informative keyjoints are unavailable or disturbed, it can compromise the efficiency and effectiveness of existing methods. This article proposes a skeleton-based model for activity recognition to deal with the occlusion problem. The main approach of the proposed model is to address occlusion by treating it as a missing value imputation problem within the feature matrix. First, it extracts features such as normalized joints, relative joint coordinates, angles, and velocity using a CNN-based technique and encodes the skeleton data into a feature matrix. The innovative use of generative adversarial imputation networks (GAIN) with bidirectional long short-term memory (Bi-LSTM), enhanced by spatial and temporal attention mechanisms, effectively handled missing data due to occlusions, ensuring a complete feature matrix for subsequent classification. Additionally, this article introduces autoencoder-based Bi-LSTM with attention modules that improve activity classification by capturing and leveraging spatial and temporal dependencies. A comprehensive experiment was conducted on both occluded and unoccluded datasets to validate the effectiveness and improvements of the proposed model. Comparative results show that the proposed model outperforms the existing state-of-the-art methods, achieving superior recognition accuracy in an occluded environment.