Multi-Master Model: Better Way to Calculate the Final Weights

Under the current age in which data privacy and security become more and more important, as the short-coming of traditional distributed machine learning algorithms, the Federated Learning algorithm (FL), became more apparent. We need a better algorithm framework to handle more complex training scenarios. The focus of this paper is on multi-master Federated Learning, specifically on the last part of how to handle the weights from different master nodes. In this algorithm, instead of using only one central server to receive all weights calculated by the clients, we use multiple master nodes, each taking care of a portion of the data. There is no interaction between clients under different master nodes. Finally, the master nodes process their weights together to get a final weight. In this paper, the number of master nodes is 3, and there are 3 approaches we propose: 1) do a simple average of the weights. 2) do a weighted average. 3) apply the traditional Fedavg algorithm on these weights. As a result, we found that for simple dataset, three approaches to get final weights in multi-master FL perform similarly and are no worse than traditional FedAvg, while for complex real-world data, approach 3 has a slightly better result.