Robust Methods for Distributed Learning

This thesis develops robust and efficient aggregation methods for distributed learning and explores their vulnerabilities. Distributed learning paradigms, such as federated and decentralized learning, enable the coordination of models across a collection of agents without the need to exchange raw data. Instead, agents compute model updates locally and share the updated models with a parameter server or their peers, followed by an aggregation step traditionally based on averaging. However, averaging-based schemes are susceptible to outliers, where a single malicious agent can significantly degrade the model's performance. This vulnerability has led to the development of robust aggregation schemes based on variations of the median and trimmed mean, which ensure robustness but at the cost of reduced efficiency. To address this drawback, efficient and robust aggregation schemes for distributed learning are developed in this thesis. We propose robust and efficient diffusion, a distributed learning framework which combines statistical efficiency with robustness. Additionally, a robust distributed Expectation-Maximization algorithm based on Real Elliptically Symmetric distributions is developed, which is highly adaptive to outliers and incorporates a robust data aggregation step, based on robust and efficient diffusion, to provide robustness against malicious agents. Simulations demonstrate the effectiveness of the proposed algorithms over non-robust methods. The motivation for studying attacks on robust aggregation schemes in decentralized learning is to understand and mitigate vulnerabilities that can significantly degrade the performance of learning algorithms. By developing and analyzing attack strategies, we can identify weaknesses in existing schemes and design more resilient and secure aggregation methods. This research enhances the robustness, reliability, and trustworthiness of decentralized learning, ensuring data integrity and improving overall security against adversarial scenarios. By leveraging the sensitivity curve, a classical tool from robust statistics, we systematically derive optimal attack patterns against arbitrary robust aggregators, rendering them ineffective in many cases. The new attack pattern, termed sensitivity curve maximization, disrupts existing robust aggregation schemes by injecting optimal perturbations. This highlights the vulnerability of classical aggregation schemes to malicious agents, leading to the degeneration or breakdown of the learning process. The effectiveness of the proposed attack is demonstrated through multiple simulations. Overall, this dissertation enhances the robustness and efficiency of aggregation methods in distributed learning while identifying and mitigating vulnerabilities through the development of optimal attack strategies.