The Many Facets of Variance Reduction in Federated Learning

Federated Learning (FL) enables clients (mobile or IoT devices) to train a shared machine learning model coordinated by a central server while keeping their data local, addressing communication and privacy concerns. In the FedAvg algorithm [2], clients perform multiple local stochastic gradient descent (SGD) steps on their datasets and send their model updates to the server. The server then aggregates these client updates to produce the new global model and sends this back to the clients for the subsequent iteration.