Toward Causal and Evidential Open-Set Temporal Action Detection

Temporal action detection (TAD) is a critical task in video understanding. Nevertheless, most existing closed-set TAD methods often struggle to replicate their high performance when completely unseen or unknown actions emerge in an open-world test environment. To this end, the open-set temporal action detection (OSTAD) task has been recently proposed to relax the closed-set TAD condition to the unknown-aware open-set detection. Given only a limited number of known action classes available in model training, precisely localizing and rejecting the unknown action instances is extremely difficult and requires strong model generalization abilities. However, existing approaches are yet far from optimal in discriminative action feature learning and prediction uncertainty estimation, which may hamper model generalization to unknown action detection. To address these issues, this paper proposes a novel Causal and Evidential Open-set Temporal Action Detection model named CEO-TAD for improved OSTAD performance. It accomplishes expressive video feature pyramid extraction, discriminative causal action feature representation learning, and reliable EDL-based prediction uncertainty estimation with our tailored network architectures and modified loss functions. Experimental results show that our proposed method achieves state-of-the-art open-set temporal action detection performance on the THUMOS14 and ActivityNet1.3 benchmarks. Ablation studies verify the effectiveness of the proposed model components.