Safe Reinforcement Learning with Lyapunov-Based Constraints for Control of an Unstable Reactor

This work presents a Lyapunov-based framework for safe reinforcement learning (RL) applied to the control of an unstable reactor. The proposed method imposes stability constraints on the value and Q-functions through a Lyapunov candidate function defined as the negative of these functions, L(s)=-V(s) and L(s,a)=-Q(s,a). Constraints enforce positivity of the Lyapunov candidate function and non-positive time derivatives, promoting monotonic behavior aligned with Lyapunov stability conditions. The framework was tested on both on-policy (PPO) and off-policy (SAC, TD3, and DDPG) RL algorithms, with performance evaluated against their baseline versions and a nonlinear Model Predictive Controller (NMPC). Results showed that stability constraints significantly improved control performance across all tested algorithms, yielding consistently higher cumulative rewards, reduced overshoot, and decreased variability. Derivative-based constraints successfully mitigated abrupt changes and oscillatory behavior in the Q and value functions, especially evident in PPO, DDPG, and TD3. This work demonstrates that Lyapunov-based constraints are an effective tool for improving the safety and stability of RL algorithms in safety-critical applications. The proposed approach avoids the complexity of auxiliary optimization and offers a computationally efficient solution for enhancing the safety and stability of RL algorithms in safety-critical control applications.