Spatio-temporal action detector with cross-stream knowledge transfer

Abstract Relying on human labor for uninterrupted monitoring through surveillance systems is highly inefficient and prone to oversight, making intelligent surveillance have great application prospects and value. To implement this technology, spatio-temporal action detectors are typically used, which are neural networks capable of locating action instances simultaneously in both time and space dimensions. To address the issue of high computational demand commonly associated with 3D backbone architectures, we propose a Dynamic Cross-Stream MovingCenter Detector (DCMOC) based on a 2D backbone network, an improved version of the MOC detector. In the DCMOC detector, the movement branch and the bounding box branch correspond to the motion and appearance features of action instances, respectively. Through a designed cross-stream knowledge transfer module, feature knowledge is transferred between the two branches to enhance feature representation. To compare with existing spatio-temporal action detectors, experiments were conducted on two datasets, JHMDB and UCF101-24. The experiments demonstrate that the DCMOC detector achieves competitive performance with significantly lower computational requirements than existing 3D architecture models and achieves optimal performance on the video-map metric at high thresholds.