Trajectory design and resource allocation for RIS-assisted UAV-enabled WPCNs using deep reinforcement learning

This paper investigates a wireless powered communication network (WPCN) facilitated by an unmanned aerial vehicle (UAV) in Internet of Things (IoT) networks, where multiple IoT devices (IoTDs) gather energy from a terrestrial energy station (ES) during the wireless energy transfer (WET) stage, followed by the UAV collecting data from these powered IoTDs with the time division multiple access (TDMA) protocol in the wireless information transfer (WIT) stage. To overcome the challenges of radio propagation caused by obstructions, we incorporate a reconfigurable intelligent surface (RIS) to enhance the link quality of the ES-IoTDs and IoTDs-UAV. The primary objective is to maximize the average sum rate of all IoTDs by jointly optimizing UAV trajectory, ES transmit power, and RIS phase shifts, along with the time allocation for WET and WIT. To this end, we reformulate the optimization problem as a markov decision process (MDP) and introduce a deep reinforcement learning (DRL) approach for addressing the formulated problem, called the proximal policy optimization (PPO) based energy harvesting with trajectory design and phase shift optimization (PPO-EHTDPS) algorithm. By continuously exploring within the environment, the PPO algorithm refines its policy to optimize the UAV trajectory, the energy phase shifts, ES transmit power, and WET/WIT time allocation. Additionally, a continuous phase shift optimization algorithm is employed to determine the information phase shifts for each IoTD to maximize average sum rate. Finally, numerical results demonstrate that the proposed PPO-EHTDPS algorithm can significantly achieve higher average sum rate and show better convergence performance over the benchmark algorithms.