A Dual-Layer Deep Reinforcement Learning Routing Approach for Integrated UAV and Satellite IoT Networks

Internet of Things Devices (IoTDs) in remote areas connect to the Internet through Low Earth Orbit (LEO) satellites. The transmission of massive IoTDs data volumes leveraging satellites faces a critical challenge, as constrained bandwidth onboard significantly impacts the performance of time-sensitive applications. Moreover, due to the limited number of satellite-ground links and the capacity of satellite access devices, it is challenging to serve all IoTDs within the satellite network coverage. To tackle these challenges, we propose a dual-layer network architecture that integrates Satellite Internet of Things (SIoT) and Unmanned Aerial Vehicles (UAVs), where UAVs serve as transmission units and satellites serve as computing units to overcome the limitations of conventional SIoT. In the dual-layer network architecture, a dual-layer routing problem is formulated to ensure the rapid transmission of tasks, and the problem is further decomposed into UAV layer routing and LEO satellite layer routing. At the UAV layer, we propose a multitask concurrent routing strategy based on task segmentation according to path bandwidth to reduce link load and improve transmission efficiency. At the LEO satellite layer, an integrated computation-transmission routing strategy is introduced to mitigate onboard bandwidth constraints through computational capabilities, thereby reducing transmitted data volume and significantly enhancing transmission efficiency. We transform the problem into a Markov Decision Process (MDP) for each layer and solve the problem using Deep Reinforcement Learning (DRL). To enhance the performance, we introduce improvements to the route algorithm. In the UAV layer, we introduce channel state averaging to reduce algorithmic complexity. In the LEO layer, we employ Prioritized Trajectory Replay (PTR) to improve learning efficiency, while a loss constraint is introduced to enhance training stability. Simulation results demonstrate that the proposed algorithm outperforms other algorithms in terms of convergence performance and overall delay.