Sign-Based Distributed Learning With Byzantine Resilience Based on Audit Mechanism

In this paper, we study the problem of distributed learning (DL) with devices transmitting sign information of the local gradients to the server under communication constraints, where the devices are susceptible to Byzantine attacks. For this problem, a sign-based gradient descent method with majority vote and stochastic 1-bit quantization (Sign-M-stochastic) has been proposed very recently. However, the Byzantine resilience of Sign-M-stochastic is inherently limited, based on the fact that all Byzantine devices and honest devices participate equally in the training process. To overcome this drawback and enhance the resilience to Byzantine attacks, inspired by the audit-based distributed detection systems, we propose a novel DL method with an audit mechanism (DL-AM). In each iteration, the sign information of the local gradients are obtained by the devices from stochastic 1-bit quantization. All devices, partitioned into groups, send the sign information to the server through multiple paths, both directly and via other devices in the same group. This approach provides the server with additional information about the identities of the devices, which enables the server to form the global model update by aggregating the sign information of different devices with varying weights. We analyze the convergence performance of the proposed method from a theoretical perspective. Finally, numerical results demonstrate the superiority of DL-AM over the baseline methods.