RoboAuditor-LLM: A Robotic System for Energy Auditing via a Large Language Model and an Actionable Semantic Feature Map

Buildings significantly contribute to electricity consumption, increasing the risk of costly blackouts and safety issues due to growing demand. However, plug loads remain underexplored in energy audits due to their complexity and variability, highlighting a critical gap in energy management practices. We propose RoboAuditor-LLM, a robotic system for automating initial steps of energy audits in indoor environments, by translating human inspectors’ instruction into executable robot actions. RoboAuditor-LLM consists of three modules: a large language model (LLM) planner for parsing human instructions and generating executable actions for robots, an Actionable Semantic Feature Map for providing spatial information of static objects and corresponding goal poses, and a Robot Toolbox for providing diverse downstream skills, including navigation, device counting, and device status verification. RoboAuditor-LLM utilizes open-source foundation models’ reasoning capacity for understanding complex human instructions, and coding ability for translating natural language into executable code for robots. We evaluate RoboAuditor-LLM in real-world institutional offices using a quadruped robot. We achieve a 76.91% average success rate for device localization and LLM-matched score of 4.01 (out of 5.00) for device status recognition.