Natural Sit-to-Stand Control of a Musculoskeletal Model with Deep Reinforcement Learning-Based Reward Shaping

Understanding the neuromuscular control mechanisms underlying the human sit-to-stand (STS) movement, a fundamental daily activity, is crucial for enhancing the physical capabilities of older adults and advancing applications in robotics and exoskeletons. This paper introduces a deep reinforcement learning (DRL) approach with reward shaping to replicate natural human STS movements using a lower limb musculoskeletal model with three degrees of freedom and nine muscles. We used proximal policy optimization (PPO) with two behavior-guided reward shaping functions,: one focused on joint angle changes (R<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">joint</inf>) and the other on head height changes (R<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">head</inf>), each tailored to the model's capabilities and the movement's objectives. To assess the effectiveness of our control method, we collected kinematic and kinetic data of human STS movements, allowing for a comparison of joint angles between the musculoskeletal model and human subjects. Both reward conditions demonstrated a strong correlation with experimental data, with the R<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">head</inf> reward achieving a correlation of 0.9 and faster convergence compared to R<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">j</inf><inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">oint</inf>. Additionally, R<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">head</inf>more accurately captured human joint dynamics, enhancing the realism of the simulated movements, while R<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">joint</inf> proved more effective in reducing energy consumption. These findings highlight the potential of DRL with reward shaping to generate realistic and efficient movement patterns, paving the way for future applications in robotics and assistive technologies.