LLM-Enhanced Mobile Robot Navigation and Scene Description for Indoor Environments

This research aims to develop an LLM-enhanced mobile robotic system for enhanced scene description. The system integrates Simultaneous Localization and Mapping (SLAM) for real time and precise environmental mapping and localization, with a Large Language Model (LLM) to provide detailed scene descriptions in natural language. This integration addresses the challenge of accurately and reliably understanding and describing complex and dynamic indoor environments. The system was developed and tested in simulation using the ROS2 framework with Gazebo Classic. Results demonstrated the LLM’s capability to describe and reason about complex scenes effectively. The current implementation is limited by single-image scene description and lack of scene memory or interactive chat. Future work will focus on enabling conversational interaction, incorporating memory, and integrating a robotic arm to expand the system’s manipulation capabilities and real-world applicability.