On Limitations of Ensuring Stability in Reinforcement Learning Under Robustifying Control

Reinforcement learning demonstrates impressive performance across domains, yet plain reinforcement learning controllers lack stability guarantees, limiting practical deployment, where safety sometimes is a critical issue. While robustifying control elements from adaptive control theory being integrated into conventional reinforcement learning controllers have been proposed as a solution, we identify fundamental limitations of this approach. We critically analyze an actor-critic framework for nonlinear continuous-time systems, proving that a prevalent robustifying control technique contains a critical flaw invalidating its claimed stability guarantees. Our analysis reveals this approach may actively destabilize systems and degrade critic learning. We establish a novel stochastic convergence theorem demonstrating how environment stability directly impacts learning performance - a connection overlooked in existing stabilization techniques. Our mobile robot case study empirically confirms these theoretical limitations, showing robustifying controls deteriorate both stabilization and critic weight convergence compared to plain actor-critic methods. These findings highlight the need f or a lternative stabilization approaches that maintain learning performance while ensuring control objectives.