Machine Learning Convergence Behavior for an Airborne Surveillance Mission

Expediting the fielding of AI/ML systems requires developing multi-agent architectures and learning protocols that ensure stability, robustness, and performance guarantees in the presence of disturbances, unknown and changing environments, and adversarial attacks. The objective of this paper is to thoroughly evaluate several Reinforcement Learning (RL) and Machine Learning (ML) protocols, including traditional Q-learning, Double Deep Q-Network (DDQN), Genetic Algorithms (GA), and hybrid ML architectures for an airborne surveillance mission modeled as a Markov Decision Process (MDP). The airborne surveillance mission MDP being evaluated contains both positive and negative rewards, along with self-loops with finite rewards that can result in improper policies and for which the boundness of the Q-learning algorithm has not been theoretically established. The performance of the different ML approaches is evaluated from the perspective of the speed of convergence to reach and maintain the optimal policy, which is equivalent to the number of samples at which the state value function error becomes zero, the observed asymptotic behavior of the actual and estimated Bellman error equations, and the convergence of a new Course of Action (CoA) state function to a steady-state fixed point that is the optimal CoA. The type of sampling, namely, asynchronous, partially synchronous, or fully synchronous performed and the learning rate, in particular, constant, linear, polynomial, and piecewise continuous, employed are shown to have a significant impact on the resultant convergence speed and performance of the learning protocol. It is demonstrated that the CoA state function is completely observable after each training iteration making it ideal for evaluating learning convergence in real-time applications.