A Multimodal Framework for Human-Centric Robotic Navigation Using LLMs, VLMs, and 2D LiDAR

This paper presents a novel multi-modal frameworkfor robust, human-centric robotic navigation by integrating LargeLanguage Models (LLMs), Vision-Language Models (VLMs), and2D LiDAR. The proposed system addresses the limitations ofpurely geometric mapping by enabling robots to construct andutilize a semantically-rich ”cognitive map” of their environment.During an initial exploration phase, a 2D LiDAR generatesa high-precision geometric map, while a VLM simultaneouslyprocesses camera feeds to generate descriptive captions andobject metadata for Points of Interest (POIs). This enriched mapserves as a structured knowledge base for a hierarchical planningsystem. The LLM, acting as a high-level planner, interpretsnatural language commands and translates them into a sequenceof actionable, semantic sub-goals by querying the cognitive map.A low-level planner then executes these commands, leveraging the2D LiDAR for precise localization and real-time obstacle avoid-ance. The framework’s modularity enables powerful emergentcapabilities, including open-vocabulary navigation and contextualunderstanding, while mitigating the computational overhead andlatency associated with a purely end-to-end approach. The paperdiscusses the system’s architecture, data representation, and apath for empirical validation using established benchmarks.