Context-aware 3D CNN for action recognition based on semantic segmentation (CARS)

Human action recognition is a prominent area of research in computer vision due to its wide-ranging applications, including surveillance, human–computer interaction, and autonomous systems. Although recent 3D CNN approaches have shown promising results by capturing both spatial and temporal information, they often struggle to incorporate the environmental context in which actions occur, limiting their ability to discriminate between similar actions and accurately recognize complex scenarios. To overcome these challenges, a novel and effective approach called Context-aware 3D CNN for Action Recognition based on Semantic segmentation (CARS) is presented in this paper. The CARS approach consists of an intermediate scene recognition module that uses a semantic segmentation model to capture contextual cues from video sequences. This information is then encoded and linked to the features captured by the 3D CNN model, resulting in a comprehensive global feature map. CARS integrates a Convolutional Block Attention Module (CBAM) that utilizes channel and spatial attention mechanisms to focus on the most relevant parts of the relevant 3D CNN feature map. We also replace the traditional cross-entropy loss with a focal loss that can better deal with underrepresented and hard- to-classify human actions. Extensive experiments on well-known benchmark datasets, including HMD51 and UCF101, show that the proposed CARS approach outperforms current 3D CNN-based state-of-the-art approaches. Moreover, the context extraction module in CARS is a generic plug-and-play network that can improve the classification performance of any 3D CNN architecture. • An effective scene action recognition based on semantic contextual information. • Integration of a reliable attention mechanism to enhance feature learning. • Improving the recognition of underrepresented action classes via focal loss. • Experiments on challenging benchmarks show higher performance over SOTA techniques.