PseR: Pseudo-Label Refinement for Point-Supervised Temporal Action Detection

Point-supervised Temporal Action Detection (PS-TAD) is an emerging research direction for label-efficient learning. Current pseudo-label-based methods have achieved satisfactory detection performance. However, the performance gap between PS-TAD and fully-supervised methods remains significant. In this paper, we attribute such a large performance gap to the poor quality of pseudo-labels. Moreover, we propose a <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Pse</b>udo-label <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</b>efinement (PseR) framework to obtain higher-quality pseudo-labels, consisting of three stages: seed proposal generation, proposal propagation, and refinement network. At the seed proposal generation stage, we use point annotations and the existing PS-TAD method to generate a pseudo-label for each point. The temporal boundaries of this pseudo-label cover the corresponding point annotation and achieve the highest confidence in the existing PS-TAD method, referred to as the seed proposal. Then, proposal propagation generates proposals with varying durations and center positions around the seed proposal through scale and center perturbations. These proposals, along with the seed proposal, form the proposal bag corresponding to the point annotation. Subsequently, within the refinement network, a selection module selects proposals within each bag close to the action instance. To further refine the selection process, a ranking module is proposed to obtain temporal confidence to assist in selecting the best proposals. Ultimately, the refinement network can generate higher-quality pseudo-labels. We conduct extensive experiments on four challenging benchmarks and demonstrate that our PseR significantly enhances the state-of-the-art PS-TAD methods, resulting in average mAP improvements of 3.7%, 3.3%, 9.3%, and 1.5% on THUMOS'14, GTEA, BEOID, and ActivityNet-1.3, respectively. Our source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zhenyingfang/PseR</uri>.