Reinforcement Learning for Building Control: Direct Actuator or PID-Mediated Control?

Building control systems typically use either setpoint regulation, where controllers determine optimal setpoints for low-level systems, or direct actuator control, which bypasses the setpoint layer. Despite widespread hierarchical approaches, empirical evidence comparing these strategies remains limited. This paper presents a systematic evaluation of both paradigms using reinforcement learning (RL) with the Building Optimization Performance Test (BOPTEST) framework. We implement and compare: (1) setpoint-trained RL agents using PPO, DQN, and A2C algorithms, (2) direct actuation-controlled RL agents with identical algorithms, and (3) a conventional PI controller baseline. Experiments span two simulation environments (BESTEST Hydronic Heat Pump and BESTEST Air systems) with constant electricity pricing. Results demonstrate that setpoint regulation achieves near-perfect thermal comfort (> 99% discomfort reduction) with modest energy cost increases (4.8–5.0%), while direct actuator control offers smaller energy penalties (1.8–3.2%) with substantial comfort improvements (> 80% discomfort reduction). Most significantly, setpoint regulation consistently demonstrated faster learning, requiring 48–79% fewer training steps across algorithms and environments. The setpoint-trained approach also exhibited 38% lower control signal variance (measured as the standard deviation of actuator commands) over the 14-day testing period. These findings provide quantitative evidence supporting a two-level control architecture, where an RL agent optimizes setpoints for existing low-level controllers, while also identifying conditions where direct actuator control by an RL agent may be advantageous. Our research contributes valuable empirical insights for designing energy-efficient building control strategies using reinforcement learning, with setpoint regulation offering significant advantages in training efficiency and control stability.