Contrastive Learning based on Counterfactual Sequences for Weakly Supervised Video Moment Localization

Video Moment Localization (VML) is the task of finding a specific temporal segment in an untrimmed video that aligns with a language query. Given the difficulty in acquiring detailed temporal annotations, weakly supervised methods—using video-level video-language alignment—have become popular. Current approaches often employ contrastive learning, which involves distinguishing positive and negative proposals and calculating cross-modal similarities. However, due to the lack of fine-grained annotations, it's challenging to effectively capture foreground details, thereby affecting proposal accuracy and reducing contrastive learning's impact. To address these challenges, we propose a method involving multi-perspective counterfactual sequence construction, aiming to enhance supervision signals and improve the contrastive learning. We create biased unimodal and unbiased multimodal sequences, use a training strategy to reduce unimodal biases, and introduce video-level and fine-grained counterfactual sequences to refine model accuracy in distinguishing important elements. Our experimental results demonstrate significant improvements in model performance.