Generating Informative Benchmarks for Reinforcement Learning

Reinforcement learning (RL) benchmarking has long relied on learning curves and cumulative reward tables, yet these metrics fail to capture critical design challenges, such as environment sensitivity, robustness, and reproducibility. This paper introduces a novel benchmarking paradigm that rigorously evaluates these understudied properties through a generative framework grounded in converse optimality. By constructing families of control problems with provably optimal solutions, our method enables statistical hypothesis testing of an algorithms response to environment perturbations both during training and post-deployment. Central to this approach is a problem generator that synthesizes parametric variations of dynamics and costs, paired with a sensitivity evaluation pipeline. To illustrate the frameworks usefulness, we apply it to compare environment sensitivity between Adam and RAdam optimizers – a case study designed to clarify how exactly the methodology provides the means to evaluate reinforcement learning algorithms in an informative way. This work contributes to RL evaluation by shifting focus from sample efficiency alone to methodologies that assess algorithms under conditions reflective of practical engineering challenges.