A single modern graphics processing unit (GPU) typically has the memory bandwidth equivalent to many central processing unit (CPU) nodes. This makes GPU hardware appealing for linear solvers that tend to require high memory bandwidth and fast inter-core communication. Reservoir simulators are designed to handle a wide range of simulation models, and to obtain peak performance the linear solver must be well suited to the resulting linear systems. This fact can lead to disappointing performance when shifting the linear solver from CPU to GPU. To fully utilize the capabilities of the latest GPU devices, we must transition from coarse-grained to fine-grained parallel preconditioners. To enable such high levels of parallelism in the linear solver a common approach is to employ a multicolor reordering of the linear system. Depending on the specific properties of the simulation model, this process can cause a significant weakening of the parallel preconditioner, resulting in much slower convergence. In some situations, this slow convergence can cause an order of magnitude increase in the linear iteration count, and result in the GPU linear solver performing worse than the CPU version. In this paper we analyze the impact on performance of employing different coloring schemes for different simulation models and we identify how the coloring can be automatically adapted for the properties of each simulation model. In this way, the performance improvements expected on the GPU can be realized for a wider range of simulations.

Commercial reservoir simulators have traditionally been optimized for parallel computations on central processing units (CPUs). The recent advances in general-purpose graphics processing units (GPUs) have provided a powerful alternative to CPU, presenting an opportunity to significantly reduce run times for simulations. Realizing peak performance on GPU requires that GPU-specific code be written, and also requires that data are laid out sympathetically to the hardware. The cost of copying data between the CPU memory and GPU memory at the time of this writing is egregious. Peak performance will only be realized if this is minimized. In paper Cao et al., 2021, the authors establish approaches to enable a simulator to give excellent performance on a CPU or GPU, with the same simulation result using either hardware. We discuss how their prototype was generalized into high-quality, maintainable code with applicability across a wide range of models. Different parts of a reservoir simulator benefit from different approaches. A modern, object-oriented simulator requires components to handle initialization, property calculation, linearization, linear solver, well and aquifer calculations, field management, and reporting. Each of these areas will present architectural challenges when broadening the scope of the simulator from CPU only to supporting CPU or GPU. We outline these challenges and present the approaches taken to address them. In particular, we discuss the importance of abstracting compute scheduling, testing methods, data storage classes, and associated memory management to a generic framework layer. We have created a high-quality reservoir simulator with the capacity to run on a CPU or GPU with results that match to within a very small tolerance. We present software engineering approaches that enable the team to achieve and maintain this in the future. In addition, we present test outcomes and discuss how to achieve excellent performance. To our knowledge, no simulator capable of both CPU simulation and full GPU simulation (meaning simulation with no copies of full grid-size data for purposes other than reporting) has been presented. We will present novel software approaches used to implement the first such commercial simulator.