A Single Human Video as a Motion Prior: Residual Reinforcement Learning and Diffusion Distillation for Vision-Based Dexterous Manipulation
Learning dexterous manipulation typically requires hundreds of demonstrations, making data collection costly and hard to scale. We propose a three-stage framework learning from a single human video. First, hand and object trajectories extracted from the video are retargeted to the robot as motion priors. Second, combining residual learning with trajectory augmentation, a privileged state policy is trained via proximal policy optimization to generalize to unseen object poses. Third, a diffusion model distills this policy into a visual controller: the privileged policy collects self-generated successful trajectories under standard and perceptual domain randomization conditions, which are blended to train the diffusion model to output action sequences from raw point clouds. Across three dexterous manipulation tasks, the privileged state policy attains high success rates with one demonstration video per object, achieving higher SR10 success rates than DexMV baselines on the same Adroit platform with two orders of magnitude fewer demonstrations, while the distilled vision-based policy operates from raw point clouds without privileged state information. This work demonstrates that effective dexterous manipulation strategies can be learned even when human supervision is limited to a single demonstration video. The video supplies the motion prior, while all the remaining training experience, including the robot rollouts that are used for distillation, is generated autonomously in simulation. This offers a practical pathway for low-cost robotic skill learning.
Authors
- 黄生传
- Yantai Huang (ORCID: https://orcid.org/0000-0002-3451-4937)
- Qingchun Jiao (ORCID: https://orcid.org/0000-0002-8695-0722)
- Zhenyu Zhang
Institutions
- Zhejiang University of Science and Technology (CN)
Publication Details
- Journal
- Electronics
- Published
- 2026-11-19
- DOI
- https://doi.org/10.3390/electronics15225403
- Primary Topic
- Robot Manipulation and Learning
- Type
- article
- Field-Weighted Citation Impact
- 0.00