Leveraging Open-Vocabulary Object Detection in Goal-Oriented Navigation for Assessing Indoor Energy–Intensive Devices

Energy auditing is crucial to improve energy efficiency in buildings. This paper focuses on the initial stage of energy auditing: collecting data on energy-intensive devices, which account for 20% of major building energy consumption. Manual data collection for these devices significantly increases the workload and duration of energy audits because it involves identifying and localizing various device types in unknown environments. Existing robotic approaches to automate this process are either not scalable to unseen objects or lack effective exploration strategies for object localization. We present a robotic system for automating the initial step of energy audits in indoor environments, aimed at efficient navigation and precise localization of energy-intensive devices. This system integrates an RGB-D SLAM (simultaneous localization and mapping) module for mapping, a Relevance Mapper for goal-oriented exploration, and a Navigator for path planning. The system uses a pretrained open-vocabulary semantic segmentation model to identify human-queried, energy-intensive devices without additional training and to optimize exploration. Tested on 12 buildings from the HM3D data set in simulation, our approach demonstrates an improved success rate in detecting targeted energy-intensive objects, achieving a 73.38% success rate, compared to a control group without Relevance Mapper, which only reached 67.38%. Notably, our approach shows an evident improvement in large-scale buildings by 12%. Furthermore, we conducted real-world tests with a quadruped robot equipped with an RGB-D camera to validate our system’s effectiveness in complex real-world environments, resulting in an 86.67% success rate. Our method offers a scalable, cost-effective solution for initial steps of energy auditing to facilitate informed decisions on building system retrofitting.