Leveraging Propagation Delays: A Delay-Aware Multi-Agent Reinforcement Learning MAC Protocol for Underwater Acoustic Networks

Underwater acoustic networks are typically distributed in nature and have been attracting much research interest recently. Such networks are characterized by long propagation delays, which pose challenges for the medium access control (MAC) protocol design in underwater acoustic networks. In this paper, instead of considering long propagation delay as a negative effect, we exploit it as an advantage. We propose a multi-agent reinforcement learning (MARL)-based MAC protocol without requiring acknowledgement feedback, named Delay-Aware Multi-Agent Reinforcement Learning Multiple Access (DA-MARLA), which leverages propagation delays to achieve higher throughput. Furthermore, the throughput achieved can exceed that of systems with zero propagation delay. In developing DA-MARLA, we introduce a novel MARL algorithm, termed delay-aware multi-agent proximal policy optimization (DA-MAPPO). Specifically, to leverage the long propagation delays, we propose two period-based mechanisms that coordinate nodes’ transmission schedules to reduce collisions and balance cooperation and competition among nodes. To ensure reliable operation, we incorporate a sequential policy update mechanism. This mechanism offers accurate performance evaluation for each node and establishes update sequences during centralized training. Simulation results show that our method consistently outperforms baseline methods across various network topologies while maintaining robustness, demonstrating that the propagation delays can be effectively utilized to enhance network efficiency and providing a new avenue for performance improvement in underwater acoustic networks.