Enabling Intelligent Reservoir Engineer: Exploring New Possibilities with Reinforcement Learning and Proxy Models

Abstract In the digital transformation of the petroleum industry, reservoir dynamic optimization confronts dual challenges: the complexity of multi-physics coupled modeling and the demand for real-time decision-making. Traditional numerical simulation methods are constrained by high computational costs, reliance on static historical data, and expert-driven experience, which limits their ability to address the dynamic evolution of reservoir properties and nonlinear interactions in heterogeneous fluids. To tackle these issues, this study introduces a physics-informed machine learning proxy model framework. By integrating deep autoencoders with neural ordinary differential equations (Neural ODEs), it constructs a high-fidelity, low-computational-cost dynamic prediction model. The framework employs deep autoencoders to nonlinearly compress high-dimensional 3D reservoir dynamic data and integrates static variables and control parameters using Neural ODEs to describe state changes. This approach significantly reduces data dimensions and enables fast prediction. Experiments conducted using the SPE9 benchmark 3D reservoir model demonstrate that the proxy model reduces the original physical field data dimensions by approximately 66.7%, validating its efficiency and reliability in dimensionality reduction and dynamic forecasting. The efficient and high-precision proxy model provides an ideal environment for reinforcement learning. Through the cooperative design of the agent action space and the reward mechanism, the closed-loop optimization of the digital twin architecture is innovatively realized, and the production parameters can be adjusted in seconds in real time. This study breaks through the timeliness bottleneck of traditional numerical simulation, and proposes an intelligent decision-making model that integrates data-driven and physical models, providing innovative solutions for intelligent reservoir development and oilfield intelligent transformation.