ACFed: Communication-Efficient & Class-Balancing Federated Learning with Adaptive Consensus Dropout & Model Quantization

Federated learning (FL) trains machine learning models over heterogeneous and decentralized datasets. Communication between client and server can be a major bottleneck for FL, especially in cases of large models. Moreover, real-world FL problems often involve data heterogeneity issues such as class imbalance. We propose an approach to address these issues of class balance and communication efficiency in Federated Learning. Our strategy is based on two key elements: 1. a novel adaptive voting-based federated dropout on client models addressing communication bottlenecks and data heterogeneity, and 2. a heterogeneous quantization method that can adjust to clients' bandwidth requirements. We conduct experiments across several datasets and models demonstrating that these two components work together to balance the trade-off between communication costs and model performance with clients having heterogeneous communication bandwidth. Importantly, our approach improves performance on imbalanced datasets like CIFAR-10-LT and CIFAR-100-LT, which is critical for addressing class imbalance in federated learning. On CIFAR-10 We get approximately a seven-factor reduction in communication cost without degrading the quality of the model.