Temporal Segment Fourier Transformer for Aciton Quality Assessment

Action Quality Assessment (AQA) plays an important role in video analysis, and unlike action recognition tasks, AQA tasks are more challenging because they require attention to subtle differences within the class. In order to capture fine-grained temporal semantic information, we designed a temporal segmentation transformer (TST) using the powerful sequence modeling capability of transformer to parse the complete action sequence into sub-action parts. The AQA methods mainly based on transformer structure usually has high computational overhead and inference time, so we designed a Fourier-based token mixer (FTM) to replace the self-attention layer, which significantly reduces computational complexity while maintaining accuracy. The experimental results on two benchmark datasets show that our FTM method effectively accelerates training and inference without significantly affecting accuracy.