Privacy-Preserving Personalized Decentralized Learning With Fast Convergence

Personalized decentralized learning aims to train individual personalized models for each client to adapt to Non-IID data distributions and heterogeneous environments. However, the distributed nature of decentralized learning is insufficient for protecting client training data from gradient leakage danger. In this paper, we investigate a <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</u>rivacy-preserving <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</u>ersonalized <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</u>ecentralized <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</u>earning optimization mechanism instead of traditional SGD. We design the P2DL mechanism to optimize our proposed objective function, whereby adjusting the regularization term parameter for a resilient local-global trade-off. Instead of exchanging gradients or models, auxiliary variables with knowledge can be transferred among clients to avoid model inversion and reconstruction attacks. We also provide theoretical convergence guarantees for both synchronous and asynchronous settings. Particularly, in case of synchronous communication, its convergence rate <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}\left ({{{}\frac {1}{k}}}\right)$ </tex-math></inline-formula> matches with the optimal result in decentralized learning, where k is the number of communication rounds. Extensive experiments are conducted to verify the effectiveness of newly proposed P2DL comparing with the state of the arts.