Provable mean-field Langevin neural actor-critic algorithms for multi-objective reinforcement learning

Abstract Actor-critic methods with theoretical convergence guarantees constitute an important class of algorithms for multi-objective reinforcement learning. To date, existing provable multi-objective actor-critic algorithms are limited to linear critic approximation and conventional stochastic actor updates. Consequently, a gap remains between the theoretical understanding of multi-objective actor-critic methods and the empirical success of deep neural actor-critic algorithms. To fill this gap, we propose a provable M ean- F ield L angevin M ulti- O bjective N eural A ctor- C ritic algorithm, referred to as MFL-MONAC, where mean-field Langevin dynamics is incorporated into both objective-wise critic evaluation and Pareto-guided actor improvement. Concretely, the critic performs regularized mean-field Langevin policy evaluation for each objective, while the actor uses MGDA-based gradient aggregation with momentum to construct a Pareto-guided update direction and applies mean-field Langevin dynamics to induce controlled particle-level exploration. We further establish a finite-time analysis for the MFL-MONAC algorithm, i.e., we show that the MFL-MONAC algorithm achieves ε -Pareto stationarity at a rate of $${{\mathcal{O}}}(1/\sqrt{T})$$ O ( 1 / T ) , where T is the number of iterations. Moreover, MFL-MONAC achieves a sufficient sample complexity of $${{\mathcal{O}}}({\varepsilon }^{-6})$$ O ( ε − 6 ) . Finally, we conduct experiments on representative MORL benchmarks to demonstrate the effectiveness of MFL-MONAC.