Locomotion Learning in a Musculoskeletal System through Reinforcement Learning of Joint Actions and LSTM Learning of Muscle Actions

The study presents an innovative method for musculoskeletal modeling that leverages reinforcement learning (RL) alongside a simplified muscle model, enabling dynamic movements like running without needing pre-recorded data or preset calculations. The key goals are to create methods for generating running actions using a half-cheetah model and to adjust for variations in muscle redundancy and stiffness. By decoupling the learning of movement patterns from the efficient use of muscles, the approach simplifies the process, making it more understandable, easier to implement, and adjustable, while also reducing computational demands. The suggested model combines an RL agent, which observes the environment and determines the necessary torque for each joint. This torque is then transformed into muscle activations via a long shortterm memory neural network (LSTM) model. Preliminary findings indicate that this combined model replicates movement patterns similar to the half-cheetah model presented by the simulation framework MujoCo and achieves a running speed of $2.4 \mathrm{~m} / \mathrm{s}$, slightly lower than the $2.9 \mathrm{~m} / \mathrm{s}$ achieved with joint motors. Although it consumes more energy, as shown by a Cost of Transport (CoT) of $\mathbf{3 . 4}$ compared to $\mathbf{1 . 5}$ for the joint motors model, the method proves to be suitable for integrating muscle functions. Future work will focus on developing a comprehensive 3D cheetah model to simulate various gaits and evaluate the scalability of this method. This will facilitate a detailed comparison between motor-driven and muscle-driven models in robotics. The main challenges to be addressed include ensuring data accuracy, optimizing parameters, and enhancing energy efficiency.