Investigating Missile Feedback Control: Reinforcement Learning, PID, or Both?

This paper evaluates the effectiveness of Deep Deterministic Policy Gradient, a type of reinforcement learning algorithm, for designing feedback controllers in missiles. Two controller configurations were developed: a black box controller that instructs a tail deflection directly to the missile, and a hybrid controller that automatically schedules PID gains by altering a set of pre-tuned nominal gains by ±50% in real time. The reinforcement learning agents observe the error in normal acceleration and the missile’s current angle of attack, pitch rate, tail deflection and tail deflection rate. The black box controller was trained to 10,000 episodes and the hybrid controller was trained to 6000, with non-zero normal acceleration step commands ranging from -5g to 5g. To assess performance and effectiveness, the step responses of both controllers and a PID controller with fixed gains were compared at setpoints of 1g, 5g and 10g. It was found that both reinforcement learning controllers met steady state requirements, however the black box controller displayed minimal overshoot but excessive controller effort, while the hybrid controller failed to manage overshoot but maintained appropriate controller effort. The hybrid controller was able to extract more performance out of the nominal gains, exhibiting clear improvements upon the rise and settling times of the initial nominal gains by 59% and 34% respectively, and displaying the potential of integrating reinforcement learning with classical control techniques.