Joint Perturbation and Aggregation for Sparse Gradients in Federated Learning

Federated learning (FL) has become increasingly popular as a privacy-preserving, distributed training approach. However, recent studies have demonstrated that sensitive information can still be inferred from FL frameworks through certain attacks. Local differential privacy (LDP) provides privacy guarantees for FL and can help reduce potential privacy breaches. Despite this, current LDP-FL frameworks suffer from utility loss because the noise injected is directly proportional to the parameter dimension. To minimize noise, recent studies have employed private selection of the Top-k dimensions of the gradient vector. Yet, these methods are limited by their perturbations to individual data points and fail to address the dimensional dependency issue inherent in LDP. To overcome this challenge, we propose a novel joint perturbation method for sparse gradients, which is divided into two stages: perturbation of the gradient index vector and assignment of gradient values. Our method perturbs high-dimensional sparse gradient vectors as a whole, rather than individual gradients, thus reducing the noise injected into model gradients while preserving the necessary level of privacy. Moreover, to counteract the impact of the randomness introduced by perturbation on model performance, we integrate the Central Limit Theorem (CLT) into the gradient aggregation process, which we call CLT-Agg. We have validated our framework using public datasets, and our findings show a significant improvement over state-of-the-art methods. Extensive experiments have confirmed the effectiveness and efficiency of our proposed framework.