Integrating Physical Effects into Action-Conditioned Frame Prediction for Robotic Manipulation

One of the core challenges in robot arm teleoperation is handling visual feedback delays, and uncertainties. Many existing approaches employ generative models and autoregressive neural networks to address this problem. However, these models are often trained on datasets that lack physical effects, such as collisions and gravity. We introduce a dataset that contains robot interactions with stacked cubes, explicitly incorporating physical effects. We propose two small-scale latent diffusion transformer models for robot manipulation conditioned on initial frames, the initial gripper position, and the gripper actions. Using our custom dataset, we evaluate our model and demonstrate its ability to predict frames involving collisions and gravity.