Federated multi-agent deep reinforcement learning for adaptive and lightweight secure protocol design in 6G networks

Abstract The move toward sixth-generation (6G) wireless networks introduces ultra-reliable low-latency communication and massive edge connectivity, but also enlarges the attack surface and demands security that is adaptive, lightweight, and decentralized. This paper presents a Federated Multi-Agent Deep Reinforcement Learning (FMADRL) framework for adaptive secure-protocol selection, in which edge nodes learn local protection policies and share only model parameters. In this scheme, each edge node functions as an autonomous reinforcement learning agent that continuously observes local network conditions—including real-time threat indicators, computational resource availability, network latency, and remaining power—and selects an appropriate security protocol configuration through a trained deep Q-network. Local training proceeds via experience replay, and model parameters rather than raw data are periodically synchronized with a central aggregator to enable collective policy improvement across the federation. We formulate protocol selection as a Markov Decision Process that balances protection utility against computation and latency cost, and we evaluate a fully reproducible implementation in which every reported number is computed from runs. On a 20,000-sample evaluation set (50.5% attack steps; oracle reward ceiling 0.528), FMADRL reaches an average reward of 0.528—matching centralized DRL (0.528) without sharing raw data—while a static protocol reaches only 0.330. Because model poisoning is a central concern for security-focused federation, we integrate and test two Byzantine-robust aggregators: under a sign-flipping attack with 30% malicious agents, standard FedAvg collapses to a policy value of − 0.513, whereas Multi-Krum and Trimmed Mean hold ≈ 0.527. Gradient compression, evaluated by counting transmitted bytes, reduces per-round communication by up to 95.5% while retaining ≥ 99.5% of policy value. The model uses only 1,380 parameters, making edge inference inexpensive. Three principal findings emerge from this evaluation: the federated policy matches centralized DRL performance without sharing any raw network data; Byzantine-robust aggregation—Multi-Krum and Trimmed Mean—sustains near-optimal policy value even when 30% of agents are actively poisoning the federation, while standard FedAvg collapses to a negative reward under identical conditions; and aggressive gradient compression cuts per-round communication by 95.5% with negligible degradation in policy quality. These results collectively confirm that a decentralized, learning-based, Byzantine-aware framework is both practical and deployable as a security foundation for 6G communications.