Dual-Reward Reinforcement Learning with Intrinsic Exploration Mechanisms for Real-Time Reservoir Management

Summary In the process of oil and gas reservoir development, the natural heterogeneity of the reservoir makes the development performance extremely sensitive to the matching of the injection-production well control scheme. Reasonable production optimization is the key to achieving balanced displacement and efficient development. Due to the complexity of oil and gas reservoirs and the large number of variables, traditional production optimization methods often face problems, such as complex and time-consuming calculations, hindering their ability to obtain global optimal solutions. Reinforcement learning (RL), with its intrinsic strengths in sequential decision-making and high-dimensional problem solving, offers a promising alternative for production optimization. Nevertheless, existing RL approaches exhibit limited exploration capabilities in complex reservoir environments, often becoming trapped in local optima. Additionally, they typically fail to incorporate well pattern information effectively, overlooking the location information and the connectivity between injection and production wells. To address these challenges, we propose a dual-reward coupled RL production optimization algorithm based on a global perspective. Specifically, the production optimization problem will be modeled as a Markov decision process. Through the intrinsic reward mechanism, the agent is encouraged to conduct deeper exploration to overcome the optimization limitations in complex environments, and the well network production information is integrated into the model in combination with the graph neural network (GNN) to enhance the modeling ability of the dynamic characteristics of the reservoir. In this way, the agent can effectively avoid relying only on local objectives for optimization and then achieve a more balanced injection and production strategy. The proposed method adaptively learns by engaging in intrinsic-extrinsic directional interactions with the uncertain reservoir environment, effectively leveraging accumulated well control experience in a manner that is similar to real-world field operating mode. Simulation results derived from two reservoir models demonstrate that, in comparison with other optimization methods, the proposed approach achieves higher net present value (NPV) and exhibits outstanding performance in oil displacement.