Learning Hyperparameter Schedules with Meta Population Based Training

Choosing optimal hyperparameters in reinforcement learning can greatly impact both the final performance and sample efficiency of an agent, as demonstrated by numerous studies on this subject in this area. Classical, static methods that seek a favorable hyperparameter configuration before training starts, aiming to keep this configuration fixed throughout the run have been traditionally employed to tackle this problem. However, adapting hyperparameters dynamically according to the training profile can often yield better results. This insight led to the field of Dynamic Algorithm Configuration (DAC). In this work, we investigate whether dynamic tuning of hyperparameters in reinforcement learning can be further improved by incorporating meta-data from related tasks. Learning hyperparameter schedules involves modeling non-stationary, complex functions. To address this challenge, we employ a deep kernel Gaussian Process, which allows us to capture intricate patterns in the hyperparameter space to guide the dynamic tuning of a parallel population of agents. In addition, we "stress-test" this approach by examining the influence of meta-data volume and the distribution of related tasks on performance. Our findings indicate that, under certain conditions, meta-learning can significantly accelerate convergence. When tasks are sufficiently related, the approach yields meaningful speedups compared to PB2, whereas when the meta-tasks are highly dissimilar, the benefits of the meta data diminish although our approach still outperforms the default static hyperparameter configurations. Furthermore, our method explores the hyperparameter space in ways that differ from PB2, highlighting a unique exploratory behavior that could offer further insights into efficient hyperparameter tuning for reinforcement learning tasks.