Integrated Online Q-Learning Design for Wastewater Treatment Processes

The efficiency and economy of the nonlinear optimal control process in wastewater treatment plants are two crucial indicators, corresponding to achieving the control objective faster and reducing the preset cost function. To accomplish this, an integrated online Q-learning (IOQL) algorithm, driven by a prior policy and an exploration policy, is proposed for nonlinear discrete-time systems characterized by nonaffine features and unknown structures. First, a prior policy based on historical or artificial experience is designed to reduce the training time of the controller and provide a more stable learning environment. By introducing a weighting factor, the impact of the prior policy on the overall learning process can be adjusted. Second, an exploration policy is trained online through new experiences collected from the real environment. By leveraging two policies, we can swiftly and smoothly adjust the critic network for approximating the cost function and the action network for approximating the exploration policy, which can gradually enhance the control outcomes. Third, a stability condition with reasonable bounds is presented for the IOQL design. Finally, experimental and comparative results pertaining to a wastewater treatment plant, specifically evaluating learning speed and cost consumption, clearly demonstrate the significant advantages and superiority of the IOQL algorithm.