Out-of-Sight Embodied Agents: Multimodal Tracking, Sensor Fusion, and Trajectory Forecasting

Trajectory prediction is a fundamental problem in computer vision, vision-language-action models, world models, and autonomous systems, with broad impact on applications including autonomous driving, robotics, and surveillance. Most existing approaches assume observations are complete and relatively clean, and thus do not adequately address out-ofsight agents or the intrinsic noise in sensing modalities (e.g., sensor measurements) caused by restricted camera coverage, occlusions, and the lack of ground-truth denoised trajectories. These factors introduce substantial safety concerns and reduce the robustness of trajectory prediction in practical deployments. In this extended study, we introduce major improvements to Out-of-Sight Trajectory (OST), a new task aimed at predicting noise-free visual trajectories of out-of-sight objects from noisy sensor observations. Based on our prior work, we expand the setting of Out-of-Sight Trajectory Prediction (OOSTraj) from pedestrians to both pedestrians and vehicles, thereby increasing its relevance to autonomous driving, robotics, and surveillance scenarios. Our improved Vision-Positioning Denoising Module utilizes camera calibration to construct a vision-position correspondence, mitigating the absence of direct visual cues while enabling effective unsupervised denoising of noisy sensor signals. Extensive experiments on the Vi-Fi and JRDB datasets demonstrate that our method achieves state-of-the-art results for both trajectory denoising and trajectory prediction, with clear gains over prior baselines. We further provide comparisons against classical denoising techniques, including Kalman filtering, and adapt recent trajectory prediction models to this setting, establishing a stronger and more comprehensive benchmark. To the best of our knowledge, this is the first work to incorporate vision-positioning projection to denoise noisy sensor trajectories of out-of-sight agents, opening new directions for future research in this area. The code and preprocessed datasets are available at https://github.com/Hai-chao-Zhang/OST.