Object-Mesh-Free Dexterous Robot Demonstrations from Human RGB-D Video and Teleoperation

Teleoperating a five-fingered robot hand to collect demonstrations is costly and scales poorly, while human video is abundant yet contains neither robot commands nor robot images. Existing approaches overlay rendered robots on human video, mainly for parallel-jaw grippers; reconstruct object meshes or point clouds in place of the original RGB; or require a wearable exoskeleton during capture. We present a pipeline that composes stationary RGB-D video of a bare human hand with teleoperation logs of the same task. The video supplies the metric hand trajectory and the scene; the teleoperation log supplies recorded finger position targets. Persistent hand-state transition alignment (PHTA) matches ordered changes in a common hand-shape representation and transfers the complete finger-command vector through one temporal correspondence, without task-phase, contact, or grasp labels. Part-aware visibility transfer replaces the human hand with the robot hand while preserving which parts were in front of or behind the object, requiring no object mesh or pose. Combining N videos with M demonstrations yields N×M composed demonstrations that reuse the M recorded finger-command trajectories across the N scenes. On four real-robot tasks, five teleoperation demonstrations composed with twenty human episodes raise behavior-cloning success from 0–45% to 50–90%, reaching in two tasks the same observed success rate as in one hundred teleoperation demonstrations.