PPO and SAC Reinforcement Learning Based Reference Compensation

In this paper, a reinforcement learning-based compensation method is used to control the UR5e robot, designed for applications such as rehabilitation. The robot is tasked with accurately reaching a fixed point in its workspace as well as performing smooth and precise tracking of predefined movement paths. Two reinforcement learning (RL) algorithms, Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) are employed as the basis for the compensator. For evaluation, a simulation of the UR5e robot, a 6-DoF industrial robot, is used to follow reference paths, such as a square path. Based on the reference in the workspace, the required position for each joint is calculated using the inverse kinematics of the UR5e. Consequently, the reinforcement learning algorithm takes the error between each joint's reference and current state and computes a correction signal. This output is then used to adjust the reference point and generate an altered error, which is processed by the proportional-derivative (PD) controller to compute the velocity control signal provided to the UR5e.