Proximal Policy Optimization with Reward Shaping for Musculoskeletal Robot Grasping

Musculoskeletal robots represent a potential breakthrough in humanoid robotics, yet the complexity of controlling musculoskeletal systems continues to pose significant challenges, restricting design and practical use. Due to the large number of muscles, the nonlinearity of muscles, and the coupling between muscles, achieving high-performance control strategies for musculoskeletal systems and robots is still difficult. This paper introduces a method for grasping tasks in musculoskeletal robots, utilizing Proximal Policy Optimization (PPO) as the core technique. By applying reward shaping, the exploration space for the grasping task is narrowed. The effectiveness of the proposed reward function in grasping tasks is validated in a simulated musculoskeletal system, achieving a task success rate of over 80%. The findings suggest that the introduced reward terms, based on both the grasping process and outcomes, significantly enhance exploration efficiency in high-dimensional action spaces, enabling effective grasping tasks with a reinforcement learning-based musculoskeletal arm while reducing energy consumption. We expect that these findings will contribute to the control and application of musculoskeletal robot grasping.