Autonomous Aerospace Systems with an Understanding of Risk and Uncertainty Using Distributional Ensemble Gaussian Mixture Model Reinforcement Learning

In recent years, autonomous vehicles are rapidly growing as an innovation in modern transportation. However, the dynamic and complex nature of real-world environments can pose a challenge to traditional control algorithms. Direct solution methods are restricted by their assumptions of linearity and/or convexity, and sampling-based methods are restricted by the time to analyze such complex environments. Reinforcement Learning (RL) provides a solution by approximating optimal control strategies without increased computation time, allowing vehicles to traverse complex environments. Despite this, RL is rarely utilized in real-world applications due to the lack of its ability to understand risk, which can pose a threat in safety-critical systems and high-risk environments. In this paper, we utilize a combination of distributional and ensemble RL to provide the agents with an understanding of risk and uncertainty. By modelling the distribution of environment rewards as a Gaussian mixture, we use risk-aware metrics to improve safety and stability. With ensemble methods we can isolate the uncertainty due to limitation in exploration and knowledge, giving agents an estimate of their situational awareness. We can improve an agent’s awareness by pushing it to explore areas it is uncertain about more and we can prevent, by simple throttling, agents from taking actions when their uncertainty is too high. Our paper will compare the methods with previous work, modelling the distribution of rewards as a quantile-based discretization, and against a baseline of soft-actor critic in a sample Unmanned Aerial System (UAS) environment.