Skeleton-Based Two-Person Interaction Recognition Using Multi-Stream Feature Fusion and Joint Evaluator

Two-person interactions are prevalent in public places, and how to recognize the friendly and violent two-person interactions is crucial for ensuring public safety. However, there are three challenges in two-person interaction recognition: the difficulty in capturing interaction relationships, the interference of interaction-irrelevant actions, and the distortion of the spatial relationship caused by camera view variations. In this paper, we propose an end-to-end multi-stream feature fusion model integrating Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) to capture interaction dynamics and perform fine-grained analysis of interactive actions. The model aims to overcome the limitations of insufficient interaction feature representation through hierarchical fusion of heterogeneous data streams. Meanwhile, to suppress interaction-irrelevant features, we propose a Mutual-Self Activity-level Evaluator (MSEr) to highlight interaction-related features that simultaneously represent individual actions and interaction semantics. Furthermore, to characterize view-invariant features, we employ the 2D Discrete Wavelet Transform (2D-DWT) to analyze action within the spatial-frequency domain. Experimental results demonstrate that the proposed method obtains competitive performance among the compared methods with accuracies of 98.3±0.2%, 94.6±0.1%, and 89.3±0.1% on the SBU-Kinect, NTU RGB+D 60, and NTU RGB+D 120 datasets respectively.