Grounding-enhanced retrieval network for temporal grounding in long videos

This paper addresses the emerging challenges in Long-form Video Temporal Grounding (LVTG). Existing methods typically follow a two-stage process: proposal retrieval followed by grounding. However, they rely solely on sentencelevel features for proposal retrieval, resulting in insufficient alignment granularity. Additionally, they fail to fully integrate grounding into the retrieval process and overlook both proposal-level and grounding-level predictions, limiting their effectiveness. Moreover, most approaches require the entire long-form video as input during training, which demands significant computational resources. To address these challenges, we propose the Grounding-Enhanced Retrieval Network (GERNet) for temporal grounding in long videos. GERNet’s key innovation is the Grounding-Enhanced Attention Pool (GEAP) module, which improves proposal retrieval by combining proposal-level and grounding-level scores for final prediction. GEAP leverages both sentence-level and token-level query information, enhancing attention pooling via frame grounding for more precise alignment between query and proposal. It also selectively uses relevant video segments for large-scale contrastive learning, reducing computational overhead. GERNet consists of two models: GER-VTG, which integrates a video temporal grounding module for more accurate grounding, and GER-RPG, which incorporates a reevaluation proposal-to-grounding module for faster grounding by selecting finer candidate proposals. Experimental results show that both models outperform existing methods, achieving state-of-the-art performance in long-form video temporal grounding.