Martingale Deep Reinforcement Learning for High-Dimensional Financial Problems

We develop a martingale-residual framework for continuous-time reinforcement learning in high-dimensional financial problems, linking policy evaluation (PE), policy gradient (PG), and entropy-regularized Actor–Critic (AC) learning. First, continuous-time PE and PG are often developed through separate constructions. We formulate a common martingale residual for the Critic and Actor, and show that, as the local time step vanishes, its rescaled policy derivative recovers the classical continuous-time PG. We also quantify how Critic approximation errors affect policy-gradient accuracy. Second, martingale-orthogonality temporal-difference (TD) methods require prescribed test functions, and their performance can depend on that choice. We introduce a weak-martingale adversarial formulation in which the test function is learned rather than prescribed. The resulting minimax problem avoids the sensitivity to hand-chosen test functions and provides a unified computational representation of the associated moment conditions. Third, direct high-dimensional Hamilton–Jacobi–Bellman (HJB) implementations can require explicit Hessians and other higher-order differential operators. Our algorithms work directly with sampled martingale residuals and first-order automatic differentiation, yielding Hessian-free implementations for both offline training from complete trajectories and online updating from sequential transitions. Numerical experiments on high-dimensional basket-option pricing and mean–variance portfolio selection demonstrate accurate PE and effective PG. In high-dimensional mean–variance portfolio-selection experiments, the proposed method achieves competitive or superior performance on several key metrics relative to benchmarks: the offline variant achieves a mean annualized return of 16.223%, the online variant attains a mean Sharpe ratio of 2.337, and fixed-budget training time increases by 19.3% as the asset dimension grows from 10 to 1000.