A Single Human Video as a Motion Prior: Residual Reinforcement Learning and Diffusion Distillation for Vision-Based Dexterous Manipulation

Learning dexterous manipulation typically requires hundreds of demonstrations, making data collection costly and hard to scale. We propose a three-stage framework learning from a single human video. First, hand and object trajectories extracted from the video are retargeted to the robot as motion priors. Second, combining residual learning with trajectory augmentation, a privileged state policy is trained via proximal policy optimization to generalize to unseen object poses. Third, a diffusion model distills this policy into a visual controller: the privileged policy collects self-generated successful trajectories under standard and perceptual domain randomization conditions, which are blended to train the diffusion model to output action sequences from raw point clouds. Across three dexterous manipulation tasks, the privileged state policy attains high success rates with one demonstration video per object, achieving higher SR10 success rates than DexMV baselines on the same Adroit platform with two orders of magnitude fewer demonstrations, while the distilled vision-based policy operates from raw point clouds without privileged state information. This work demonstrates that effective dexterous manipulation strategies can be learned even when human supervision is limited to a single demonstration video. The video supplies the motion prior, while all the remaining training experience, including the robot rollouts that are used for distillation, is generated autonomously in simulation. This offers a practical pathway for low-cost robotic skill learning.

Authors

Institutions

Publication Details

Journal
Electronics
Published
2026-11-19
DOI
https://doi.org/10.3390/electronics15225403
Primary Topic
Robot Manipulation and Learning
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A Single Human Video as a Motion Prior: Residual Reinforcement Learning and Diffusion Distillation for Vision-Based Dexterous Manipulation

黄生传, Yantai Huang, Qingchun Jiao, Zhenyu Zhang
Electronics
Robot Manipulation and Learning
article

A Single Human Video as a Motion Prior: Residual Reinforcement Learning and Diffusion Distillation for Vision-Based Dexterous Manipulation

黄生传, Yantai Huang, Qingchun Jiao, Zhenyu Zhang
article en

Abstract

Learning dexterous manipulation typically requires hundreds of demonstrations, making data collection costly and hard to scale. We propose a three-stage framework learning from a single human video. First, hand and object trajectories extracted from the video are retargeted to the robot as motion priors. Second, combining residual learning with trajectory augmentation, a privileged state policy is trained via proximal policy optimization to generalize to unseen object poses. Third, a diffusion model distills this policy into a visual controller: the privileged policy collects self-generated successful trajectories under standard and perceptual domain randomization conditions, which are blended to train the diffusion model to output action sequences from raw point clouds. Across three dexterous manipulation tasks, the privileged state policy attains high success rates with one demonstration video per object, achieving higher SR10 success rates than DexMV baselines on the same Adroit platform with two orders of magnitude fewer demonstrations, while the distilled vision-based policy operates from raw point clouds without privileged state information. This work demonstrates that effective dexterous manipulation strategies can be learned even when human supervision is limited to a single demonstration video. The video supplies the motion prior, while all the remaining training experience, including the robot rollouts that are used for distillation, is generated autonomously in simulation. This offers a practical pathway for low-cost robotic skill learning.

ElectronicsVol. 15(22)
Zhejiang University of Science and Technology (CN)
Robot Manipulation and Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon