Distributed Optimization via Mixing ADAM and Quasi-Newton Methods

The heterogeneous computational capacities of devices always negatively impact the performance of distributed optimization algorithms. In this paper, we propose a novel distributed optimization algorithm based on a mixture of different local optimizers, termed as FedMix, which has the potential to leverage the heterogeneity to enhance the performance. Specifically, FedMix is designed in the parameter-server framework, and each client can be equipped with either an Adam optimizer or a quasi-Newton optimizer for local updates based on its computational capability, which leads to a mixture of Adam and Quasi-Newton in our algorithm. As different local optimizers demand different computational capabilities, our algorithm FedMix can achieve better alignment between algorithms and computing resources, facilitating improved performance especially in the heterogeneous computational environment. Sufficient experiments are presented to substantiate the superior performance of our algorithm by comparing with existing algorithms.