Model-Based Reinforcement Learning for Chemical Dosing Optimization in a Municipal Wastewater Treatment Plant: A Comparative Study of Three Actor–Critic Algorithms

Chemical dosing control in wastewater treatment plants (WWTPs) necessitates a dynamic equilibrium between effluent compliance and operational costs, exemplifying a typical multi-objective sequential decision problem. Given the significant operational and compliance risks associated with online trial-and-error methods in full-scale plants, this study introduces a model-based reinforcement learning (MBRL) framework aimed at optimizing dosing. Utilizing 227 historical operation records, we construct a multilayer perceptron (MLP) virtual WWTP that maps 15 process states and 2 dosing actions to 6 effluent indicators, thereby providing a secure training environment for policy development. A composite reward function is formulated to incorporate penalties for effluent quality, constraints for abnormal conditions, and terms related to chemical costs. Three actor–critic algorithms—Soft Actor–Critic (SAC), Twin Delayed Deep Deterministic Policy Gradient (TD3), and Proximal Policy Optimization (PPO)—are trained for 50,000 steps and evaluated over 50 test episodes against both a random baseline and a practically deployed proportional feedforward control baseline. All three reinforcement learning methods yield significantly higher rewards: SAC achieves a mean score of 70.23 (95% CI: [67.94, 72.52]), TD3 scores 70.33 (95% CI: [68.02, 72.64]), and PPO scores 68.78 (95% CI: [66.47, 71.09]), compared to 61.01 (95% CI: [58.23, 63.80]) for proportional control and 45.87 (95% CI: [41.29, 50.45]) for random dosing. Notably, both off-policy agents (SAC and TD3) demonstrate statistically equivalent, state-of-the-art control performance (paired Wilcoxon, p = 0.0631) and converge towards highly economical, low-dosage strategies. This study validates the feasibility of data-driven virtual commissioning for WWTP dosing optimization and supports the advancement of hybrid intelligent control systems that incorporate mechanistic constraints. In addition to algorithm comparisons, this framework can be further refined to serve as a practical, data-driven decision support tool for wastewater treatment plant (WWTP) operators. It enables them to formulate daily chemical dosing plans while adhering to compliance and cost constraints.