Evolution Strategy-Assisted TD3 for Continuous Control: Policy-Update Analysis and Plant-State Boundedness

Exploration quality and policy parameter updates influence the training of deterministic actor–critic controllers. This study presents ES–TD3, an evolution strategy-assisted Twin Delayed Deep Deterministic Policy Gradient framework that combines population-based actor-parameter search with off-policy learning. Candidate actors undergo Gaussian perturbation, episodic evaluation, elite selection, and fitness-weighted recombination. The mathematical analysis establishes that the recombined parameter vector lies within the convex hull of the elite actors and bounds its displacement from the current population center by the largest elite displacement, with a perturbation-dependent bound for Gaussian candidates. For a mass–spring–damper plant with positive mass, damping, and stiffness, bounded-input/bounded-state and Lyapunov analyses establish uniform ultimate boundedness of the physical state under the bounded deterministic actor output. These guarantees concern individual recombination steps and plant-state boundedness, not asymptotic tracking or global training convergence. MATLAB/Simulink simulations compare ES–TD3 with standard TD3 and PID control, examine reference and initial condition variations, and assess sensitivity to evolutionary settings. The best observed configuration reaches the moving-average reward threshold of 126,500 at generation 203 and attains a final moving-average reward of 126,512. The reported ES–TD3 run exhibits smoother reward progression than the TD3 baseline, while the deployed policy produces bounded tracking responses across the tested scenarios. The results support the feasibility of combining evolutionary policy search with actor–critic learning for the considered control problem; broader performance claims require independent repeated trials.