Accelerated Deep Reinforcement Learning in Subsurface Reservoir Production Optimization

Summary Constrained production optimization in subsurface reservoirs under geological uncertainty presents major computational and methodological challenges. High-fidelity simulations are too slow for training reinforcement learning (RL) agents or running iterative gradient-based optimizations. Ensuring constraint satisfaction amid uncertain reservoir dynamics requires carefully designed strategies. In this study, we introduce a hybrid optimization framework that integrates a scalable multimodel embed-to-control and observe (E2CO-lite) surrogate, deep reinforcement learning (DRL) with adaptive constraint-aware reward shaping, and sequential quadratic programming (SQP) with stochastic gradients (StoSAG). The E2CO-lite surrogate accelerates simulation while preserving geological realism, enabling efficient DRL training across clustered realizations. The trained DRL agent provides generalizable control strategies to warm-start SQP, combining DRL’s exploratory strength with SQP’s constraint-handling rigor. We validate the framework on the Brugge benchmark reservoir. Proxy-based DRL training achieves more than 50X speedup compared with full-physics simulation. The DRL-SQP hybrid improves convergence and optimization outcomes over SQP with naïve initialization. The results show that DRL-assisted initialization achieves higher net present value (NPV) while satisfying constraints. Final control strategies tested on the true Brugge model confirm the method’s scalability, efficiency, and practical relevance for subsurface reservoir optimization under uncertainty.