Finding Approximate Correlated Equilibrium for Decentralized Control

We address the problem of finding an “optimal” decentralized control strategy, defined as achieving approximate Correlated Equilibrium, a generalization of Nash equilibrium. Each controller uses a Reinforcement Learning algorithm within a Markov game framework, abstracting continuous states into discrete ones. Controllers observe the system’s discrete state and incurred costs, updating their strategies privately. We prove convergence to approximate Correlated Equilibrium. Our algorithm is applied to a water distribution network with two controllers managing a tank’s water level in a decentralized setting, using real-world consumption data. Previously, this problem was deemed to be computationally challenging owing to the discrete controls (number of pumps switched on), resulting in a Mixed Integer optimization. The designed controllers aim to ensure stability, safety, and energy minimization. By training over multiple episodes, optimal weighting parameters for objectives are determined using Blackwell’s Approachability theorem. Simulation results show the attainment of approximate correlated equilibrium in accordance with our theoretical results.