Temporal Map-Based Boundary Refinement Network for Video Moment Localization

Video moment localization has gradually become a hot research problem in video understanding. Despite much remarkable progress, there are still two limitations in the following aspects. Firstly, previous methods usually directly regarded the moment candidate with the highest confidence score as the final localization result; they overlooked the inevitable deviations between moment candidates and the ground truth. Secondly, past models have not considered the problem that moment candidates with various qualities have different impacts on model training. Therefore, this paper proposes a novel Temporal Map-based Boundary Refinement Network to solve the above problems. Specifically, besides the conventional confidence scores’ prediction network, we introduce a boundary refinement network based on the 2D temporal map, which can fine-tune the temporal boundaries of generated moment candidates to obtain more precise results. Additionally, to discriminately treat diverse moment candidates with different qualities and further boost the localization performance, we devise an innovative weighted refinement loss to guide the model to focus more on refining processes of those moment candidates closer to the ground truth. Finally, we evaluate our model on two publicly available datasets, and the results of extensive experiments show that our technique outperforms the state-of-the-art methods (e.g., for the ActivityNet Captions dataset, a relative improvement of 4.63% on [email protected]).