Bridging vision-language reasoning and learned control for legged navigation in the wild
Autonomous navigation in diverse real-world environments remains one of the greatest challenges in robotics. It demands both robust low-level motion control to handle diverse terrain and high-level reasoning to interpret environmental cues. We present ROAM (Robust Open-world Autonomous Mobility), a system that enables legged robots to perform kilometer-scale navigation in complex alpine and urban settings. ROAM combines the world knowledge of a vision-language model (VLM), pretrained on internet-scale data, with robust locomotion and local planning policies learned in simulation via reinforcement learning (RL). The VLM provides common-sense reasoning and semantic interpretation of the scene, assessing risks, interpreting signs, and producing semantic labels with associated navigation costs. An open-vocabulary segmentation module converts these labels into a dense, real-time risk map, which conditions two pretrained RL policies responsible for agile locomotion and local navigation: effectively coupling high-level VLM reasoning with low-level physical control. Real-world experiments showed the system's ability to autonomously navigate a wide range of complex real-world environments. The robot completed kilometer-long hikes with minimal intervention, interpreted signs to avoid danger, and evaluated its environment to identify safe areas on which to step. This work represents a step toward the reliable deployment of systems in unstructured environments. However, it also underscores the challenges that remain in VLM reliability and spatial reasoning, indicating that foundational models are a crucial component but not yet a complete solution for navigation in the wild.
Authors
- Julian Nubert (ORCID: https://orcid.org/0000-0001-8949-6134)
- Joonho Lee (ORCID: https://orcid.org/0000-0002-5072-7385)
- Takahiro Miki (ORCID: https://orcid.org/0000-0001-8556-2819)
- Matthias F. Mueller (ORCID: https://orcid.org/0000-0001-5249-8734)
- Marco Hutter (ORCID: https://orcid.org/0000-0002-4285-4990)
- Robin Schmid (ORCID: https://orcid.org/0009-0005-1235-8901)
- Turcan Tuna (ORCID: https://orcid.org/0000-0001-8662-4890)
- Manthan Patel (ORCID: https://orcid.org/0000-0002-9171-2139)
- Julia Richter (ORCID: https://orcid.org/0009-0001-5477-6440)
Institutions
- ETH Zurich (CH)
- Apple (United States) (US)
Publication Details
- Journal
- Science Robotics
- Published
- 2026-11-18
- DOI
- https://doi.org/10.1126/scirobotics.aee1549
- Primary Topic
- Robotic Locomotion and Control
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- Horizon Therapeutics
- Alzheimer Nadační Fond