Cumulative Stabilization Probabilities During PPO Learning: A Paired Reward Study for Cart-Pendulum Swing-Up

This simulation study investigates how the cumulative probability of successful cart–pendulum swing-up and sustained stabilization changes with training progress and physical evaluation time. The target quantity, H(g,t), measures the probability that a controller has completed its first qualifying stabilization interval by time t at training generation g. Two reward formulations are compared using 100 paired training seeds, 1,000 PPO rollout/update generations per controller, and full 20-second evaluations every ten generations. The simulator uses fully nonlinear uniform-rod dynamics. Success requires a simultaneous two-second hold within specified cart-position, cart-velocity, upright-angle and angular-velocity tolerances. At the final generation, 68/100 bounded-reward controllers and 52/100 quadratic-reward controllers succeeded by 20 seconds. The observed paired difference was 16 percentage points, with a conservative 95% interval of −3.461 to +34.184 points. Quadratic achieved substantial success earlier during training but subsequently regressed. Final completion-time curves crossed, showing that neither reward uniformly dominated. The comparison is a follow-up using an already observed bounded-reward reference arm. Failures, censoring and learning regressions are retained. Supporting material provides event data, source code, configurations and reproducible analysis.