Distributed Adaptive Softmax Policy Gradient Method with Variance Reduction: A Global Convergence Analysis

The distributed policy gradient (PG) method has become a mainstream technique for finding optimal policies in multi-agent reinforcement learning. However, global convergence guarantees, which are the most basic measures of theoretical convergence performance, are rarely used in existing distributed policy gradient methods. Moreover, the convergence behaviors of existing methods are sensitive to the selection of learning rates, which leads to limited convergence speeds. To tackle this challenge, we present a new distributed adaptive softmax policy gradient algorithm called DASPG by incorporating Adam-type updates and variance reductions into a vanilla distributed policy gradient method. Moreover, we establish global convergence guarantees for DASPG, i.e., an ϵ-optimal policy can be found with a sample complexity of O˜(ϵ−4.5), which is on par with leading centralized PG methods in terms of sample efficiency. Finally, we present experimental results to demonstrate the efficacy of the proposed algorithm.