Transformer-Enhanced Physics Informed Deep Reinforcement Learning for Quadrotor Navigation

Abstract Unmanned Aerial Vehicles (UAVs) are increasingly deployed across civilian and military domains due to their mobility, flexibility, and low energy consumption. Traditional UAV path-planning and navigation techniques often suffer from high computational cost, limited adaptability, and suboptimal path generation, motivating the use of learning-based solutions. Deep Reinforcement Learning (DRL) enables UAVs to learn optimal navigation strategies directly from interactions with the environment, handling continuous and high-dimensional action spaces typical of UAV dynamics. However, conventional DRL methods struggle to capture long-term temporal dependencies. This work presents a goal-conditioned DRL framework for multirotor navigation toward a target destination, enhanced with a Transformer-encoder and a Long Short-Term Memory (LSTM) layer. Particularly, both architectures address the DRL limitations by modeling long-range dependencies through a self-attention mechanism and preserving temporal dependencies. The framework integrates two stable, probabilistic DRL algorithms, PPO and A2C. Compared with their standalone counterparts, the Transformer-enhanced architectures improve validation stability and trajectory consistency, with the most pronounced improvement observed for PPO. Furthermore, a novel reward function for continuous action spaces is proposed, combining multiple contributions with a physics-inspired term based on the Least Action Principle (LAP), which encourages efficient and dynamically consistent flight. A simulation pipeline is developed to evaluate the approach. Training is first performed on a quadrotor dynamic model, followed by Software-in-the-Loop (SIL) on PX4 and Gazebo validation and Hardware-in-the-Loop (HIL) deployment on a Raspberry-Pi (RPI). The proposed pipeline is designed to facilitate model prototyping while supporting a structured progression toward real-world flight testing. The results obtained from the simulated experiments show that the Transformer-enhanced architecture improves trajectory robustness compared to baseline DRL policies, while the proposed reward formulation promotes smoother and more action-consistent flight behavior. These findings demonstrate the effectiveness of combining transformer-based temporal modeling with physics-informed reward design for autonomous UAV navigation.