Action-conditioned latent Kalman filtering for robust spatio-temporal state estimation of deformable object

Abstract Robust spatio-temporal state estimation from noisy observations is a critical challenge in robotic manipulation. Current latent space representations derived from dynamic point clouds often suffer from severe perceptual noise and occlusions, leading to discontinuous trajectories and pronounced temporal jitter. To mitigate these artifacts, we propose an Action-Conditioned Latent Kalman Filter (AC-LKF) framework. Leveraging a hybrid architecture of PointNet++ and Gated Recurrent Units (GRU), our method models non-linear temporal dynamics within a low-dimensional latent manifold. Instead of claiming to fully model the physical causality of deformable-object dynamics, AC-LKF incorporates the robot end-effector displacement as a low-dimensional, observable, and task-relevant kinematic prior to condition the latent state transition. Furthermore, a heteroscedastic latent filtering module adaptively estimates process and measurement uncertainties, balancing the action-conditioned prior with degraded visual inputs under noise and occlusion. Evaluations using quantitative temporal metrics and reconstruction errors, supplemented by principal component analysis (PCA) and t-distributed stochastic neighbor embedding (t-SNE) visualizations, show that AC-LKF improves latent trajectory smoothness, information retention, and dynamic robustness by reducing abrupt observation-induced state transitions. These results indicate that kinematic action conditioning can improve the temporal consistency and robustness of learned geometric state representations for interactive deformable-object manipulation.