Deep Reinforcement Learning for Generalizable Field Development Optimization

Abstract

Summary The optimization of field development plans (FDPs), which includes optimizing well counts, well locations, and the drilling sequence is crucial in reservoir management because it has a strong impact on the economics of the project. Traditional optimization studies are scenario specific, and their solutions do not generalize to new scenarios (e.g., new earth model, new price assumption) that were not seen before. In this paper, we develop an artificial intelligence (AI) using deep reinforcement learning (DRL) to address the generalizable field development optimization problem, in which the AI could provide optimized FDPs in seconds for new scenarios within the range of applicability. In the proposed approach, the problem of field development optimization is formulated as a Markov decision process (MDP) in terms of states, actions, environment, and rewards. The policy function, which is a function that maps the current reservoir state to optimal action at the next step, is represented by a deep convolution neural network (CNN). This policy network is trained using DRL on simulation runs of a large number of different scenarios generated to cover a “range of applicability.” Once trained, the DRL AI can be applied to obtain optimized FDPs for new scenarios at a minimum computational cost. While the proposed methodology is general, in this paper, we applied it to develop a DRL AI that can provide optimized FDPs for greenfield primary depletion problems with vertical wells. This AI is trained on more than 3×106 scenarios with different geological structures, rock and fluid properties, operational constraints, and economic conditions, and thus has a wide range of applicability. After it is trained, the DRL AI yields optimized FDPs for new scenarios within seconds. The solutions from the DRL AI suggest that starting with no reservoir engineering knowledge, the DRL AI has developed the intelligence to place wells at “sweet spots,” maintain proper well spacing and well count, and drill early. In a blind test, it is demonstrated that the solution from the DRL AI outperforms that from the reference agent, which is an optimized pattern drilling strategy almost 100% of the time. The DRL AI is being applied to a real field and preliminary results are promising. Because the DRL AI optimizes a policy rather than a plan for one particular scenario, it can be applied to obtain optimized development plans for different scenarios at a very low computational cost. This is fundamentally different from traditional optimization methods, which not only require thousands of runs for one scenario but also lack the ability to generalize to new scenarios.

A systematic review of data science and machine learning applications to the oil and gas industry

“…Dynamic data includes ultimate drainage area and initial production rate for offset wells. He et al (2021) developed a methodology to optimize the field development plans (FDPs). This includes optimizing well counts, well locations and the drilling sequence.…”

Deep Reinforcement Learning for Constrained Field Development Optimization in Subsurface Two-phase Flow

“…In our recent work He et al [3], we developed a deep reinforcement learning technique for the field development optimization in two-dimensional subsurface single-phase flow settings. The deep reinforcement learning technique allows for the training of an artificial intelligence agent that provides a mapping from the current state of a two-dimensional reservoir model to the optimal decision (drill/do not drill and well location) in the next step of the development plan.…”

“…The deep reinforcement learning technique allows for the training of an artificial intelligence agent that provides a mapping from the current state of a two-dimensional reservoir model to the optimal decision (drill/do not drill and well location) in the next step of the development plan. Our goal in this work is to extend the procedures in He et al [3] for the field development optimization, in the presence of operational constraints, in both two-and three-dimensional subsurface two-phase flow. Once properly trained, the artificial intelligence agent should learn the field development logic and provide optimized field development plans instantly for different field development scenarios.…”

“…He et al [3] applied the deep reinforcement learning technique to the more challenging field development optimization problem in which the decisions to be made include the number of wells to drill, their locations, and drilling sequence. In contrast to the previous studies discussed, He et al [3] used DRL to develop artificial intelligence which could provide optimized field development plans given any reservoir description within a predefined range of applicability.…”

“…In contrast to the previous studies discussed, He et al [3] used DRL to develop artificial intelligence which could provide optimized field development plans given any reservoir description within a predefined range of applicability. In addition, unlike in previous studies [27,28] in which the artificial intelligence agent is represented by fully connected neural networks, He et al [3] utilized a convolutional neural network to represent the agent which allows for better processing of spatial information. The field development optimization problem in He et al [3], however, considers a single-phase flow in twodimensional reservoir models.…”