SAFormer: An Efficient Hierarchical Transformer Network Specialized for Temporal Action Detection

Temporal action detection (TAD) is a critical task of multimedia video understanding, focusing on accurately predicting the starting and ending times of action instances along with their classification. Most previous works have relied on two-stage methods such as action proposals or anchor windows to identify actions in TAD. In this work, we aim to explore a straightforward transformer-based model for localizing actions within untrimmed videos. Our proposed solution, Spatial Aware Transformer (SAFormer), introduces a one-stage action detector by utilizing a novel mechanism in self-attention. This approach embeds global channel-wise feature responses, allowing lower network layers to utilize information from a broader receptive field. Additionally, we introduce the Classification Refinement Modification Loss (CRML) to handle imbalanced action distributions. CRML incorporates a focal loss function that adjusts the loss weight for challenging samples. Leveraging these enhancements, SAFormer demonstrates robust performance on THUMOS14 (+0.3% average mAP over prior works) and ActivityNet 1.3 (36.6% average mAP). Furthermore, our method achieves state-of-the-art performance on EPIC-Kitchens 100, with an average mAP improvement of over 0.94 absolute percentage points.