Optimizing Wastewater Treatment Control Using Policy-Constrained Offline Reinforcement Learning and Lyapunov Density Model

Wastewater treatment plants (WWTPs) constitute a crucial component of contemporary societal infrastructure, playing a critical role in purifying and recycling wastewater from both municipal and industrial sources. The effectiveness and reliability of control strategies in these processes are paramount. This paper presents an offline reinforcement learning (RL) controller enhanced with the Lyapunov density model (LDM) to solve the optimal tracking control problem in wastewater treatment processes. The offline RL algorithm employs off-policy learning to derive the optimal control policy from the offline dataset without risky online trials. By incorporating the density estimation of the offline dataset, the LDM estimates the probability that the system adheres to the distribution of the dataset under the learned policy. The introduction of LDM helps constrain the learned policy, thereby enhancing its reliability when encountering unknown states. Additionally, a weighting scheme, based on the coefficient of variation, is applied to determine the relative weights of the LDM and the critic in the update of the control policy. Simulations conducted on a nonlinear system and the wastewater treatment process demonstrate that the proposed method learns a more effective policy from the offline dataset, leading to reduced implementation costs.