Neural Operator based Reinforcement Learning for Control of first-order PDEs with Spatially-Varying State Delay

Control of distributed parameter systems affected by delays is a challenging task, particularly when the delays depend on spatial variables. The idea of integrating analytical control theory with learning-based control within a unified control scheme is becoming increasingly promising and advantageous. In this paper, we design a integrated control strategy combining PDE backstepping and deep reinforcement learning (RL) for an unstable first-order hyperbolic PDE with spatially-varying delays. This method eliminates extra constraint on the delay function required for the backstepping design. We embed a DeepONet, trained to learn the backstepping controller, into a soft actor-critic (SAC) framework as a feature extractor for both the actor and critic networks. Simulation results demonstrate that the proposed algorithm outperforms standard SAC in reducing steady-state error and surpasses the backstepping controller in mitigating overshoot.