Reinforcement Learning-Based Control Framework for Wastewater Treatment Process Aeration Control

Wastewater treatment requires precise control of processes that are difficult to accurately predict and model. In particular, aeration control, a process that typically accounts for 50 percent or more of a typical wastewater plant's energy consumption, has been identified as a target area for improvement. Therefore, process improvements that lead to increased efficiency in the aeration phase of treatment will lead to significant energy savings for the entire treatment process. In this study, a reinforcement learning (RL)-based control framework is considered. This work includes development and testing, with comparison to a conventional proportional-integral-derivative (PID) controller via a representative simulation-based case study. To facilitate development and testing of the proposed control architecture, a simulated environment that is developed for wastewater treatment studies (i.e., Benchmark Simulation Model 1, BSM1) is used in this work. Two approaches for RL-based aeration control are explored: (a) using an RL agent to set the control parameters of a PID controller, and (b) using an actor-critic type RL agent to make control actions directly. In either case, the controller's manipulated variable is the aeration rate (KLa) in Tank 5 of the simulated BSM1 plant. The goal for the controller is to maintain effluent total nitrogen below permitted limits, while minimizing energy consumption. The results from this study provide insight regarding the viability of the proposed RL-based control configurations for wastewater treatment process control (e.g., the RL-PID configuration showed a 16% reduction in effluent violations), leading to potential improvements in treatment quality and energy-efficiency.