FeDLRT: Federated Dynamical Low-Rank Training

Abstract. We propose a federated dynamical low-rank training (FeDLRT) scheme to reduce client compute and communication costs—two significant performance bottlenecks in horizontal federated learning. Our method builds upon dynamical low-rank time integration schemes for manifold-constrained optimization to create a global low-rank basis of network weights, which enables client training on a small coefficient matrix. This global low-rank basis allows us to incorporate a variance correction scheme and prove global loss descent and convergence to a stationary point on a low-rank manifold. FeDLRT features dynamic augmentation and truncation of the low-rank bases to optimize computing and communication resource utilization. Notably FeDLRT only trains a small coefficient matrix per client. We demonstrate the efficiency of FeDLRT in an array of computer vision benchmarks with both independent and identically distributed (IID) and non-IID data and show a reduction of client compute and communication costs by up to an order of magnitude with minimal impacts on global accuracy. FeDLRT performs as well as classical methods such as FedAvg and FedLin, with a fraction of the memory and compute requirements.