Reinforcement Learning-Driven Prosthetic Hand Actuation in a Virtual Environment Using Unity ML-Agents

Modern myoelectric prostheses remain difficult to control, particularly during rehabilitation, leading to high abandonment rates in favor of static devices. This highlights the need for advanced controllers that can automate some motions. This study presents an end-to-end framework coupling deep reinforcement learning with augmented reality (AR) for prosthetic actuation. A 14-degree-of-freedom hand was modeled in Blender and deployed in Unity. Two reinforcement learning agents were trained with distinct reward functions for a grasping task: (i) a discrete, Booleann reward with contact penalties and (ii) a continuous distance-based reward between joints and the target object. Each agent trained for 3 × 107 timesteps at 50 Hz. The Booleann reward function performed poorly by entropy and convergence metrics, while the continuous reward function achieved success. The trained agent using the continuous reward was integrated into a dynamic AR scene, where a user controlled the prosthesis via a myoelectric armband while the grasping motion was actuated automatically. This framework demonstrates potential for assisting patients by automating certain movements to reduce initial control difficulty and improve rehabilitation outcomes.