Language-guided Robust Navigation for Mobile Robots in Dynamically-changing Environments

In this paper, we develop an embodied AI system for human-in-the-loop navigation using a wheeled mobile robot. We propose a direct yet effective method for monitoring the robot’s current plan to detect changes in the environment that significantly impact the intended trajectory of the robot and then query a human for feedback. We also develop a means to parse human feedback expressed in natural language into local navigation waypoints and integrate it into a global planning system by leveraging a map of semantic features and an aligned obstacle map. Extensive testing in simulation and physical hardware experiments with a resource-constrained wheeled robot tasked to navigate in a real-world environment validate the efficacy and robustness of our method. This work can support applications such as precision agriculture and construction, where persistent monitoring of the environment provides users with information about the state of the environment.