Addressing the Sim-to-Real Gap in Reinforcement Learning for UAVs by Recovering Markovian Properties with Applications to Moving Window Traversal

Abstract Recent research has witnessed a surge in the use of reinforcement learning (RL) as a control design paradigm for unmanned aerial vehicles (UAVs). This trend is driven by advances in AI and the increasing complexity of UAV applications. Despite these developments, a primary challenge in adopting RL for UAV control is the sim-to-real gap, where mismatches between training models and real-world deployment can undermine the expected performance. This paper introduces a novel RL framework that enables direct transferability of policies from simulation to real-world platforms by explicitly addressing the non-Markovian nature of UAV dynamics in the presence of state and action latency. We attribute the non-Markovian behavior to the partial observability of the dynamics caused by system delays in sensors and actuators. We address partial observability and thereby recover the Markovian properties of the system by explicitly modeling those delays and accounting for them in model identification and RL training. We experimentally validated the proposed RL framework by traversing a UAV through a narrow moving window. The traversal maneuver was achieved solely through offline data-based training without post-training fine-tuning. The proposed delay-aware controller achieved successful traversal in all 8 deployment trials, whereas the non-augmented baseline failed in all 4 deployment attempts, highlighting the importance of delay-aware design for reliable sim-to-real transfer. A video of the real-world experiments is available at the following link: https://www.youtube.com/watch?v=_tcyHzMf-5c .