Reinforcement Learning for Building Control: Direct Actuator or PI-Mediated Control?

Building control systems typically use either setpoint regulation, where controllers determine optimal setpoints for low-level systems, or direct actuator control, which bypasses the setpoint layer.Despite widespread hierarchical approaches, empirical evidence comparing these strategies remains limited.This paper presents a systematic evaluation of both paradigms using reinforcement learning (RL) with the Building Optimization Performance Test (BOPTEST) framework [1].We implement and compare: (1) setpoint-trained RL agents using PPO [8], DQN [6], and A2C [5] algorithms, (2) direct actuation-controlled RL agents with identical algorithms, and (3) a conventional PI controller baseline.Experiments span two simulation environments (BESTEST Hydronic Heat Pump and BESTEST Air systems) with constant electricity pricing.Results demonstrate that setpoint regulation achieves near-perfect thermal comfort (>99% discomfort reduction) with modest energy cost increases (4.8-5.0%), while direct actuator control offers smaller energy penalties (1.8-3.2%) with substantial comfort improvements (>80% discomfort reduction).Most significantly, setpoint regulation consistently demonstrated faster learning, requiring 48-79% fewer training steps across algorithms and environments.The setpointtrained approach also exhibited 38% lower control signal variance (measured as the standard deviation of actuator commands) over the 14-day testing period.These findings provide quantitative evidence supporting a two-level control architecture, where an RL agent optimizes setpoints for existing low-level controllers, while also identifying conditions where direct actuator control by an RL agent may be advantageous.Our research contributes valuable empirical insights for designing energy-efficient building control strategies using reinforcement learning, with setpoint regulation offering significant advantages in training efficiency and control stability.