MANIPDREAMER: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidance

Recent advancements in robotic manipulation video synthesis have shown promise, but challenges remain in achieving robust instruction-following and high visual fidelity. Existing approaches like RoboDreamer rely on linguistic decomposition for compositional understanding but often overlook sequential dependencies and key visual cues such as depth and semantics, which are crucial for improving visual quality. We propose ManipDreamer, an advanced world model based on an action tree structure and enriched with visual guidance. Each instruction is represented as a structured action tree, with nodes linked to learned embeddings that improve compositional reasoning. These embeddings condition the world model, enhancing instruction-following accuracy. To improve visual fidelity, we integrate depth and semantic information via a visual guidance adapter compatible with the world model. ManipDreamer thus enhances both instruction-following accuracy and visual realism. Evaluations on standard benchmarks show ManipDreamer outperforms RoboDreamer, increasing PSNR from 19.55 to 21.05, SSIM from 0.7474 to 0.7982, and reducing Flow Error from 3.506 to 3.201 on unseen tasks. It also boosts the average success rate in robotic manipulation tasks by 2.5% across six RLbench tasks.