Evolutionary Dynamics of Stochastic Q Learning in Multi-Agent Systems

Since high complexity and uncertainty is inherent in real-world environments that can influence the strategies choices of agents, we introduce a stochastic perturbation term to characterize the interference caused by uncertain factors on multi-agent systems (MASs). Firstly, the stochastic Q learning is designed by introducing stochastic perturbation term into Q learning, and the corresponding replicator dynamic equations of stochastic Q learning are derived. Secondly, we focus on two-agent games with two and three action scenarios, analyzing the impact of learning parameters on agents’ strategy selection and demonstrating how the learning process converges to its Nash equilibria. Finally, we also conduct a sensitivity analysis on exploration parameters, demonstrating how exploration rates affect the convergence process in potential games. The analysis and numerical experiments offer insights into the effectiveness of different exploration parameters in scenarios involving uncertainty.