Retrieval-Augmented Task-Aware Transformer for Video Procedure Learning

Procedure learning aims to jointly recognize and localize key steps from instructional videos given an ordered step list. The goal is to predict precise temporal spans for each step. Existing methods typically treat this as a sequence-to-sequence alignment task. However, as the provided step lists lack deeper task-level knowledge, we observe that these models often struggle with two challenges: (1) Missing cross-route task generalization, where models overfit to single annotated pattern and fail to capture the diverse yet semantically valid routes of the same task; and (2) Insufficient temporal-visual grounding, where the absence of concrete timing and appearance priors in the provided step lists leads to boundary drift and step error when predicting on individual videos. To address these challenges, we propose the Task-Aware Transformer (TAT), a unified framework that integrates task-level priors into the alignment process. TAT consists of two complementary components: (i) Task-Aware Structured Memory Construction, which aggregates semantic, temporal, and visual knowledge from multiple execution routes into a structured memory. To ensure the memory remains faithful to supervision, we introduce a Memory Constraint Loss that jointly encodes semantic consistency and temporal order. (ii) Retrieval-Augmented Video-Step Alignment, which retrieves a knowledge-enriched route from the memory to serve as contextual guidance, injecting fine-grained priors into the frame-to-step alignment. We further design two alignment strategies, a lightweight cross-modal decoder and a Vision-Language Model association, to balance efficiency and generalization. Extensive experiments on four benchmarks (YouCookII, COIN, CrossTask, and ProceL) demonstrate that TAT consistently outperforms existing methods, particularly in generalizing to novel procedural compositions, achieving state-of-the-art performance.

Authors

Institutions

Publication Details

Journal
IEEE Transactions on Pattern Analysis and Machine Intelligence
Published
2026-11-20
DOI
https://doi.org/10.1109/tpami.2026.3755542
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Retrieval-Augmented Task-Aware Transformer for Video Procedure Learning

Dejie Yang, Minghang Zheng, Yang Liu
IEEE Transactions on Pattern Analysis and Machine Intelligence
Multimodal Machine Learning Applications
article

Retrieval-Augmented Task-Aware Transformer for Video Procedure Learning

Dejie Yang, Minghang Zheng, Yang Liu
article en

Abstract

Procedure learning aims to jointly recognize and localize key steps from instructional videos given an ordered step list. The goal is to predict precise temporal spans for each step. Existing methods typically treat this as a sequence-to-sequence alignment task. However, as the provided step lists lack deeper task-level knowledge, we observe that these models often struggle with two challenges: (1) Missing cross-route task generalization, where models overfit to single annotated pattern and fail to capture the diverse yet semantically valid routes of the same task; and (2) Insufficient temporal-visual grounding, where the absence of concrete timing and appearance priors in the provided step lists leads to boundary drift and step error when predicting on individual videos. To address these challenges, we propose the Task-Aware Transformer (TAT), a unified framework that integrates task-level priors into the alignment process. TAT consists of two complementary components: (i) Task-Aware Structured Memory Construction, which aggregates semantic, temporal, and visual knowledge from multiple execution routes into a structured memory. To ensure the memory remains faithful to supervision, we introduce a Memory Constraint Loss that jointly encodes semantic consistency and temporal order. (ii) Retrieval-Augmented Video-Step Alignment, which retrieves a knowledge-enriched route from the memory to serve as contextual guidance, injecting fine-grained priors into the frame-to-step alignment. We further design two alignment strategies, a lightweight cross-modal decoder and a Vision-Language Model association, to balance efficiency and generalization. Extensive experiments on four benchmarks (YouCookII, COIN, CrossTask, and ProceL) demonstrate that TAT consistently outperforms existing methods, particularly in generalizing to novel procedural compositions, achieving state-of-the-art performance.

IEEE Transactions on Pattern Analysis and Machine IntelligenceVol. PP
Peking University (CN)
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Retrieval-Augmented Task-Aware Transformer for Video Procedure Learning — Dejie Yang, Minghang Zheng, et al. · IEEE Transactions on Pattern Analysis and Machine Intelligence (2026) | TGRS Research Map | TGRS