From Simulation to Reality: A Study of Reinforcement Learning Control in Operational Building Environments

Abstract Deploying reliable and cost-effective heating, ventilation, and air conditioning (HVAC) control strategies is essential for modern buildings. While reinforcement learning (RL) has shown promise in simulation, its real-world effectiveness remains underexplored. This study presents one of the first end-to-end field evaluations of an RL-based HVAC controller trained entirely in a data-driven simulation environment. The optimal policy is first trained using a data-driven digital twin of an office building. This trained policy is then deployed across two air handling units (AHUs) in the building under two different scenarios: one with static and the other with dynamic thermal comfort limits. Field results in the static case show a successful transfer from simulation to real world, where the RL controller consistently maintains thermal comfort and achieves comparable energy use to the baseline building management system (BMS). In contrast, RL performance degrades under dynamic comfort conditions, revealing its sensitivity to nonstationary environments and real-world complexities. To unpack this, we present a detailed analysis identifying key contributing factors, such as reward design vulnerabilities, policy generalization limits, and the sim-to-real gap, which can serve as a reference for future deployments. This work provides empirical validation and critical insights into the opportunities and current limitations of RL-based HVAC control in real-world buildings.