Reinforcement learning-based drone simulators: survey, practice, and challenge

Abstract Recently, machine learning has been very useful in solving diverse tasks with drones, such as autonomous navigation, visual surveillance, communication, disaster management, and agriculture. Among these machine learning, two representative paradigms have been widely utilized in such applications: supervised learning and reinforcement learning. Researchers prefer to use supervised learning, mostly based on convolutional neural networks, because of its robustness and ease of use but yet data labeling is laborious and time-consuming. On the other hand, when traditional reinforcement learning is combined with the deep neural network, it can be a very powerful tool to solve high-dimensional input problems such as image and video. Along with the fast development of reinforcement learning, many researchers utilize reinforcement learning in drone applications, and it often outperforms supervised learning. However, it usually requires the agent to explore the environment on a trial-and-error basis which is high cost and unrealistic in the real environment. Recent advances in simulated environments can allow an agent to learn by itself to overcome these drawbacks, although the gap between the real environment and the simulator has to be minimized in the end. In this sense, a realistic and reliable simulator is essential for reinforcement learning training. This paper investigates various drone simulators that work with diverse reinforcement learning architectures. The characteristics of the reinforcement learning-based drone simulators are analyzed and compared for the researchers who would like to employ them for their projects. Finally, we shed light on some challenges and potential directions for future drone simulators.