A belief-based multi-agent reinforcement learning approach for electric vehicle coordination in a residential community

This paper proposes a decentralized learning-based control scheme for solving the charging/discharging coordination problem of multiple electric vehicles (EVs) operating in a residential community, with a point of common coupling, a time-of-use tariff rate, and the uncertainty in the daily routine of the EV owners. In the proposed methodology, we model the interaction of the EV owners as a Markov Game and approach the problem through the lens of Multi-agent reinforcement learning (MARL). To this end, we propose a novel Belief-based learning algorithm (BBQL), based on the distributed training and decentralized execution architecture. In BBQL, while training the agents form a belief about their neighbours (connected over a communication network) and simultaneously use it to better approximate their optimal action-value function. As for testing, each agent acts in a "pure" decentralized manner by playing its best-response to the developed belief. The proposed algorithm holds merit in addressing the typical key-aspects of MARL algorithms, i.e., scalability, privacy and fairness. Finally, we perform numerical simulations by using "real" historical demand, photovoltaic generation, electricity tariff and EV specification data. We compare the proposed BBQL with state-of-the-art Independent Learners. In particular, BBQL achieves a better trade-off between minimizing the electricity bill and maximizing the satisfaction of the range anxiety. Speaking statistically, BBQL results in increase in the number of days the range anxiety constraint is satisfied (with reduction in the standard deviation), while simultaneously reducing the average bill of the households by (with reduction in the standard deviation).