A Comprehensive Evaluation of Byzantine Attacks and Defenses Under Data Heterogeneity

In this work, we present an in-depth overview of recent advances in Byzantine-resilient distributed learning, focusing on both defense strategies and the threat model. While considerable efforts have been made to develop robust aggregations, the underlying threat model, especially under non-IID data distributions, remains under-explored. This imbalance may create a false sense of security about the effectiveness of current defenses. To address this gap, we calibrate established Byzantine attacks to better reflect the challenges of leaning on heterogeneous data, enabling more realistic stress testing of defenses. Through systematic evaluation on standard benchmark datasets and using diverse partitioning strategies, we show that data heterogeneity provides adversaries with a larger leeway for model poisoning. We leverage this insight to critically evaluate existing defenses. Our findings underscore the need to assess robustness not only through defense design, but also via carefully constructed and calibrated threat models. Additionally, we provide actionable recommendations to improve evaluation practices and outline key research directions to advance the field.