Privacy Meets Performance: Enhancing Distributed Simulation-based Federated Multi-agent Learning with Privacy-preserving Surrogate Model

In recent years, model-free multi-agent reinforcement learning (MaRL) has become a powerful tool for learning effective policies to solve optimization problems. However, individual agents may raise concerns about sharing their internal data, simulation models, and decision models in collaborative optimization. Distributed simulation (DS) and federated learning have been widely used as privacy-preserving methods to hide simulation details and maintain data and model privacy. Despite their benefits, these methods often require large amounts of interaction and data to converge, which leads to a high communication time, especially if the agents are distributed around the world. To address this issue, we propose a distributed surrogate model for DS-based federated MaRL to utilize the surrogate model instead of DS during the training. This can enhance data efficiency and effectiveness to accelerate agent learning while maintaining data and model privacy. An aerospace supply chain (SC) is used as the experimental scenario to evaluate the performance of our proposed approach, in terms of SC profits, training convergence, and execution time. Experimental results show that our proposed approach can achieve higher SC profits with the same number of simulation runs, converge faster, and reduce execution time to gain the same level of SC profits.