Transformer-Based Physics Informed Proximal Policy Optimization for UAV Autonomous Navigation

During the last two decades, Unmanned Aerial Vehicles (UAVs) have been employed for a wide range of civil and public domain applications, as well as in missions to Mars. In complex autonomous exploration scenarios, particularly in GPS-denied environments, the software integrated into the Guidance, Navigation, and Control (GNC) system plays a critical role in ensuring UAV stability and autonomy. To meet these requirements and address the limitations of traditional navigation techniques, the development of Deep Reinforcement Learning (DRL) approaches to support decision-making tasks has gained significant traction in recent years. The goal of the paper is twofold: i) to present a comparison between the traditional Proximal Policy Optimization (PPO) and the augmented PPO with a transformer architecture, ii) to achieve smooth and efficient trajectories by designing a continuous physics informed reward function accounting for the Least Action Principle (LAP). The results demonstrate that PPO achieves significantly improved performance when integrated with the transformer, as well as high efficiency of the trained agent when simulating a specific flight path. This enhancement highlights the potential of transformer-based architectures to more effectively address complex decision-making tasks than traditional DRL methods.