Reinforcement learning algorithms for continuous-time VWAP targeting execution

The optimal execution problem has always been a continuously focused research issue, and many reinforcement learning (RL) algorithms have been studied. In this paper, we address the execution problem of targeting the volume-weighted average price (VWAP) and propose a relaxed stochastic optimization framework with an entropy regularizer to promote exploration. We derive the explicit formula for the optimal policy, which follows a Gaussian distribution with its mean value being the solution to the original problem. By extending the framework of continuous RL to processes with jumps, we provide theoretical proofs for the convergence and performance of RL algorithms in this new setting. Additionally, we design two Actor-Critic algorithms, called ML-AC and MO-AC. First, minimizing the martingale loss function yields the optimal estimation of critic network parameters in the mean-square sense. Second, we utilize the martingale orthogonality condition as an alternative approach. The convergence of all algorithms has been verified across different environments, demonstrating a larger advantage in the environment with a stronger price impact. RL algorithms do not rely on explicit model assumptions or parameter estimation, enabling them to learn directly from interactions with the environment.