Retrieval-Augmented Task-Aware Transformer for Video Procedure Learning
Procedure learning aims to jointly recognize and localize key steps from instructional videos given an ordered step list. The goal is to predict precise temporal spans for each step. Existing methods typically treat this as a sequence-to-sequence alignment task. However, as the provided step lists lack deeper task-level knowledge, we observe that these models often struggle with two challenges: (1) Missing cross-route task generalization, where models overfit to single annotated pattern and fail to capture the diverse yet semantically valid routes of the same task; and (2) Insufficient temporal-visual grounding, where the absence of concrete timing and appearance priors in the provided step lists leads to boundary drift and step error when predicting on individual videos. To address these challenges, we propose the Task-Aware Transformer (TAT), a unified framework that integrates task-level priors into the alignment process. TAT consists of two complementary components: (i) Task-Aware Structured Memory Construction, which aggregates semantic, temporal, and visual knowledge from multiple execution routes into a structured memory. To ensure the memory remains faithful to supervision, we introduce a Memory Constraint Loss that jointly encodes semantic consistency and temporal order. (ii) Retrieval-Augmented Video-Step Alignment, which retrieves a knowledge-enriched route from the memory to serve as contextual guidance, injecting fine-grained priors into the frame-to-step alignment. We further design two alignment strategies, a lightweight cross-modal decoder and a Vision-Language Model association, to balance efficiency and generalization. Extensive experiments on four benchmarks (YouCookII, COIN, CrossTask, and ProceL) demonstrate that TAT consistently outperforms existing methods, particularly in generalizing to novel procedural compositions, achieving state-of-the-art performance.
Authors
- Dejie Yang
- Minghang Zheng (ORCID: https://orcid.org/0000-0003-1612-975X)
- Yang Liu (ORCID: https://orcid.org/0000-0002-4259-3882)
Institutions
- Peking University (CN)
Publication Details
- Journal
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2026-11-20
- DOI
- https://doi.org/10.1109/tpami.2026.3755542
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- article
- Field-Weighted Citation Impact
- 0.00