Hierarchical reinforcement learning for enhancing stability and adaptability of hexapod robots in complex terrains

In the field of hexapod robot control, the application of central pattern generators (CPG) and deep reinforcement learning (DRL) is becoming increasingly common. Compared to traditional control methods that rely on dynamic models, both the CPG and the end-to-end DRL approaches significantly simplify the complexity of designing control models. However, relying solely on DRL for control also has its drawbacks, such as slow convergence speed and low exploration efficiency. Moreover, although the CPG can produce rhythmic gaits, its control strategy is relatively singular, limiting the robot’s ability to adapt to complex terrains. To overcome these limitations, this study proposes a three-layer DRL control architecture. The high-level reinforcement learning controller is responsible for learning the parameters of the middle-level CPG and the low-level mapping functions, while the middle and low level controllers coordinate the joint movements within and between legs. By integrating the learning capabilities of DRL with the gait generation characteristics of CPG, this method significantly enhances the stability and adaptability of hexapod robots in complex terrains. Experimental results show that, compared to pure DRL approaches, this method significantly improves learning efficiency and control performance, when dealing with complex terrains, it considerably enhances the robot’s stability and adaptability compared to pure CPG control.