On the Design of Reward Function for Reinforcement Learning–Based Trajectory Tracking Control Using the TD3 Algorithm

Reward design is one of the most influential factors in determining the performance of reinforcement learning (RL) algorithms in continuous-control tasks. This paper presents a comparative study of three reward formulations for training a TD3 agent to perform three-dimensional trajectory tracking using a particle kinematic system modeled as double-integrator dynamics. The first reward is a linear formulation that assigns proportional penalties based on normalized position and velocity tracking errors. The second reward employs an exponential decay based on the same error terms to emphasize high-precision tracking and reduce sensitivity to smaller deviations. The third reward, a Positive-bonus Quadratic Reward (PQR), combines quadratic penalties on position and velocity errors with a positive constant bonus, forming a hybrid structure that balances guidance and corrective feedback. All three reward functions are evaluated across three representative trajectories—vertical ascent, straight-line cruise, and helical motion—and analyzed with respect to convergence behavior, and tracking accuracy. The results show that the exponential reward achieves higher tracking accuracy across the three evaluated trajectories, while the PQR formulation penalizes the agent heavily, leading to less stable learning. These findings provide practical insights into how reward shaping influences the behavior of TD3-based trajectory tracking agents and offer guidelines for designing effective reward functions in continuous-control RL applications within the tested setting.