Video Action Recognition Based on Two‑stream Feature Enhancement Network

[Purposes] Two-stream convolutional networks primarily achieve high recognition accuracy by fusing spatial and temporal features of videos. Traditional two-stream convolutional networks extract temporal features by using optical flow or temporal shift modules. The disadvantage of using optical flow lies in the heavy workload, while the disadvantage of using temporal shift modules is that they may damage the original videos’ spatial and channel features. [Methods] In this paper, a two-stream network called the Two-stream Feature Enhancement Network (TFEN) was proposed to address these issues. To solve the problem of feature damage caused by temporal shift, a Spatial Enhancement-Temporal Shift Module (SE-TSM) and a Channel Enhancement-Temporal Shift Module (CE-TSM) were proposed to enhance features after each time shift, for improving damaged features. For the issue with optical flow, a fusion-based frame difference network was proposed for extracting temporal features, which was less time-consuming and easier to operate when compared with optical flow. To address the weak motion information in frame differences, a Sports Improvement Module (SIM) was introduced to enhance motion features and improve performance. [Results] The network in this work achieves 96.1% and 75.7% accuracy on the public video datasets UCF101 and HMDB51, respectively, which are superior to current mainstream networks.