DNA‐Former: Video DNA‐Aware Transformer for Human Action Recognition

ABSTRACT Human action recognition is achieved by extracting category‐based common semantic features in videos. However, most of the current transformers draw attention to the space‐time relationship of visual patches, with limited ability to perceive semantic connections between them. We notice that the visual expression of videos comes from semantics, which is similar to the expression mechanism of human deoxyribonucleic acid (DNA). To mitigate the considerable gap between video visual features and semantic information, we build a sparse video DNA space by mimicking the human DNA, and present a video transformer termed as DNA‐Former. Specially, we design a DNA dual‐attention block to learn the correlation between video DNA information and visual patches in the space‐time dimension. In this block, we explore the cross‐shaped window self‐attention, which is rarely applied in video transformers. To further select more expressive visual patch features, we develop the contrastive disentangler head to filter out the DNA‐aware useless features based on contrastive learning. Our proposed DNA‐Former is performed on different benchmarks (i.e. Kinetics‐400, Something‐Something V2, UCF‐101 and HMDB‐51) with comprehensive ablation studies and visualization analysis. Experimental results demonstrate the effectiveness of our method, showing favourable accuracy against state‐of‐the‐art methods.