Decentralized Federated Learning with Auto-switchable Local Optimizers

In this paper, we propose a novel decentralized learning algorithm over networks, termed as DLAGD, which combines the consensus mechanism with an auto-switchable local optimizer. Specifically, each node updates its local model parameter based on either gradient descent algorithm or Adam-like algorithm, which depends on a local switching function. Such switchable local optimizer can better capture the structure information of local loss function than existing algorithms relying on a fixed local optimizer, then has the potential to accelerate the convergence rate and improve the generalization performance. We theoretically prove that DLAGD can converge to the neighborhood of a stationary point at a sublinear rate for non-convex problems. Finally, numerical experiments are presented to show the superior performance of DLAGD in terms of the training rate and test accuracy.