Design of a PPO-PID Controller Based on Reinforcement Learning

This study proposes an adaptive PID controller algorithm, which develops through an enhanced Proximal Policy Opti-misation (PPO) method. The controller employs a data-driven framework, using a reinforcement learning agent to autonomously explore and optimise the PID gain parameters. In the design of the state space, the system error <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$e(t)$</tex>, the first order difference of the output <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\Delta y(t)$</tex> and the second order difference <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\Delta^{2}y(t)$</tex> are combined to form the input feature vector of the system. To enhance the efficiency of network training, the input states undergo an online normalization preprocessing step. The reward function is designed as a comprehensive mechanism, integrating multiple objectives namely, tracking accuracy, overshoot re-duction and oscillation suppression to ensure both smoothness and efficiency in the system's responses. Numerical simulations on a representative nonlinear system confirm the effectiveness of the proposed method. Experimental results demonstrate that PPO-PID substantially enhances tracking accuracy and stability compared to conventional PID control, underscoring its superior performance in nonlinear control applications.