Decentralized deep learning in statistically heterogeneous environments

In modern machine learning, the dominant approach to training models relies on centralized datasets. However, this paradigm is often impractical or even prohibited in real-world scenarios. Concerns about data privacy, ownership, and the ethical use of publicly available data are rapidly growing, especially with increasing scrutiny on how personal data is handled. Furthermore, collecting, storing, and managing large-scale datasets incurs substantial costs. For instance, millions of smartphone users generate vast amounts of data daily -- photos, sleep patterns, text messages, and more -- which is expensive and often infeasible to process centrally. In response, distributed machine learning has emerged as a promising alternative. Distributed machine learning trains models across multiple users without centralizing data, addressing privacy concerns and logistical challenges. In this framework, data remains with clients, who train models locally and share model updates instead of data. A prominent example is federated learning, which uses a central server to coordinate training by aggregating and distributing updates. In contrast, decentralized learning removes the central server, enabling clients to communicate directly in a peer-to-peer network. However, significant data variability across clients -- data heterogeneity -- complicates model aggregation and reduces performance. This thesis proposes novel strategies to improve decentralized learning, focusing on client collaboration and data heterogeneity. First, it introduces peer-selection and clustering techniques, enabling clients to collaborate selectively with peers whose data distributions are similar. This approach circumvents the limitations of a single global model, which may fail to generalize well across diverse clients. Second, the thesis develops privacy-preserving methods to estimate data similarity and strengthens user privacy using multi-armed bandits, enabling dynamic, adaptive collaboration among clients. Beyond addressing static data heterogeneity, the thesis also tackles the challenge of evolving data distributions. New algorithms are proposed to enable models to adapt over time, ensuring robust performance even as client data distributions change. The research further extends these methods to generative models, presenting a novel ensemble approach for training generative adversarial networks (GANs) in distributed settings. Overall, the contributions of this thesis advance the scalability, efficiency, and privacy of distributed machine learning systems. By enhancing these systems' ability to manage diverse data environments, the work ensures more reliable and personalized model performance across clients, paving the way for broader applications of distributed machine learning.