Model-Based Policy Optimization for Legged Locomotion

In this paper, we explore model-based reinforcement learning (MBRL) for legged locomotion in robotics, employing state-of-the-art techniques to evaluate the performance and stability of the learning process under different models and learning strategies. Our algorithm utilizes policy search methods, specifically leveraging deep ensembles to construct a dynamic model and integrating the soft actor-critic (SAC) framework for policy learning. Deep ensembles are traditionally employed to address model uncertainties and improve the robustness of the learning process, however, this relationship is barely evaluated in practice. We conduct a comprehensive comparison of different architectural designs and ensemble strategies within our dynamic model on several simulated legged robot platforms. Our evaluation includes benchmarking the performance of different strategies of state-of-the-art algorithms, providing a detailed analysis of learning speed, stability, and overall effectiveness. The results highlight the importance of certain design criteria which are typically assumed as granted. This study provides valuable insights into the application of deep ensembles and SAC in MBRL for legged locomotion and complex robotic control systems.