Hierarchical Curriculum Learning with Optimized Experience Replay for Sample-Efficient Humanoid Locomotion

We propose a hierarchical reinforcement learning framework that synergistically integrates curriculum learning, diversity-optimized experience replay, and physics-guided reward design to address the dual challenges of sample inefficiency and dynamic balancing in bipedal locomotion. Our methodology features three key innovations: 1) A phased curriculum learning strategy that progressively escalates environmental complexity from static terrain adaptation to dynamic perturbation resistance and multimodal terrain generalization; 2) A partitioned experience replay system with diversity-aware buffer management using dual-criteria prioritization (TD-error thresholds and trajectory similarity metrics); 3) Physics-informed reward engineering with smooth transition mechanisms between dense foot-trajectory tracking rewards and sparse velocity-based rewards. Experimental validation on the Joyson Bipedal Robot demonstrates 14.9% improvement in final reward metrics compared to baseline RL. The framework achieves robust locomotion under significant external perturbations and adapts to untrained terrains with 97.5% motion fidelity. Our modular architecture establishes new benchmarks for sample-efficient humanoid control while maintaining real-time responsiveness (100Hz policy execution, 1000Hz PD control), showing significant potential for deployment in unstructured environments requiring safe human-robot interaction.