An evaluation of nature-inspired meta-heuristic algorithms for hyperparameter optimization in reinforcement learning

The problem of effective hyperparameter tuning remains a significant problem in Reinforcement Learning (RL). The learning stability and the overall convergence speed in tabular Q-learning and Deep Q-learning methods is highly contingent on the learning parameters (hyperparameters) tuning correctly. In this study we proposed the use of three recent Nature-Inspired Metaheuristic Algorithms (NIMAs): Aquila Optimizer (AO), Manta Ray Foraging Optimization (MRFO) and the Chameleon Swarm Algorithm (CSA), to find the appropriate Q-learning hyperparameters (α, γ, ε) through combinations using five different benchmark environments; FrozenLake-v1, Taxi-v3, CliffWalking-v1, MountainCar-v0, and CartPole-v1. Through a set of experiments, the optimizers were compared based on the assessed cumulative reward, overall learning stability, convergence behaviour and time to solve. CSA was determined to be the optimiser with the best performance in a repeated demonstration in stochastic and more complex environments due to its strong and stable learning dynamics. In an environment with an underlying structure, AO was determined to converge quicker than other methods and provided lower computational expense when implementing. MRFO in the sampled experiments appeared to be advantageous for tasks with delayed rewards that were sparse. The overall findings from the study reinforce the importance of the selection of an optimiser in reinforcement learning, and demonstrate that CSA has the largest utility for performance improvements across RL experiments.