VARUNA: verifiable adaptive resilient unified framework against byzantine attacks in federated learning-based IDS

Abstract The integration of Federated Learning (FL)-based Intrusion Detection Systems (IDS) in Internet of Things (IoT) faces significant challenges due to the statistical heterogeneity of distributed datasets. The IDS datasets are often highly imbalanced and biased toward majority classes, leading to degraded IDS performance. To mitigate this, Generative Adversarial Network (GAN) models are deployed at central server nodes to balance datasets and maintain system heterogeneity. However, the security of FL systems remains vulnerable to adversarial attacks, which can manipulate model updates and compromise system integrity. This research introduces the Multi-Agent GAN Network Exploitation Tactic (MAGNET), leveraging actor-critic-based reinforcement learning to manipulate GAN models. MAGNET distributes these compromised models to selected clients, enabling the simulation of complex adaptive attacks. Additionally, we propose the Coordinated Model Attack Network System (CMANS), which disrupts global model convergence through malicious gradient manipulation techniques such as Gaussian noise injection, gradient scaling, and inversion. These attacks significantly impact the performance of FL environments applied to IDS datasets. We present a robust Verifiable Adaptive Resilient Unified (VARUNA) Framework for trustworthy IDS in resilient FL systems to address these vulnerabilities. This approach employs an adaptive trust score to detect and eliminate Byzantine activities, effectively neutralizing over 99% of malicious clients in scalable FL environments. VARUNA achieves consistent error rate reductions of up to 9.4% across all attack classes, restoring model performance from an average error increase of 0.03 under CMANS and MAGNET attacks back to the pre-attack baseline across all four aggregation methods (FedAvg, Krum, Trimmed Mean, and Median). Our solution ensures a secure, efficient, and trustworthy IDS, outperforming undefended FL baselines by a statistically significant margin on all three evaluated benchmark IDS datasets.