Parallelizing and optimizing large‐scale 3D multi‐phase flow simulations on the Tianhe‐2 supercomputer

Abstract

Summary The lattice Boltzmann method (LBM) is a widely used computational fluid dynamics method for flow problems with complex geometries and various boundary conditions. Large‐scale LBM simulations with increasing resolution and extending temporal range require massive https://en.wikipedia.org/wiki/High-performance_computing#High-performance computing (HPC) resources, thus motivating us to port it onto modern many‐core heterogeneous supercomputers like Tianhe‐2. Although many‐core accelerators such as graphics processing unit and Intel MIC have a dramatic advantage of floating‐point performance and power efficiency over CPUs, they also pose a tough challenge to parallelize and optimize computational fluid dynamics codes on large‐scale heterogeneous system. In this paper, we parallelize and optimize the open source 3D multi‐phase LBM code openlbmflow on the Intel Xeon Phi (MIC) accelerated Tianhe‐2 supercomputer using a hybrid and heterogeneous MPI+OpenMP+Offload+single instruction, mulitple data (SIMD) programming model. With cache blocking and SIMD‐friendly data structure transformation, we dramatically improve the SIMD and cache efficiency for the single‐thread performance on both CPU and Phi, achieving a speedup of 7.9X and 8.8X, respectively, compared with the baseline code. To collaborate CPUs and Phi processors efficiently, we propose a load‐balance scheme to distribute workloads among intra‐node two CPUs and three Phi processors and use an asynchronous model to overlap the collaborative computation and communication as far as possible. The collaborative approach with two CPUs and three Phi processors improves the performance by around 3.2X compared with the CPU‐only approach. Scalability tests show that openlbmflow can achieve a parallel efficiency of about 60% on 2048 nodes, with about 400K cores in total. To the best of our knowledge, this is the largest scale CPU‐MIC collaborative LBM simulation for 3D multi‐phase flow problems. Copyright © 2015 John Wiley & Sons, Ltd.

Pore-Scale CO2 Displacement Simulation Based on the Three Fluid Phase Lattice Boltzmann Method

“…The porosity and permeability of the core sample used in Li et al 121 are 7.5% and 200 nD, respectively, whereas those used in Vega and Brien 42 are 35% and 1.3 mD, respectively. The final oil recovery from CO 2 -EOR in Li et al 121 and Vega et al 40 are nearly 67 and 100%, respectively. In this study, the porosity and permeability of the used sample are closer to those of Vega and Brien.…”

“…62−71 Eshraghi et al 72 reported that the LBM model could be used in the simulation of dendritic solidification with 36 billion grid points, and the parallel efficiency was almost 100% using 1000−4068 CPU cores. Li et al 40 reported that the threedimensional (3D) multiphase LBM simulation cases (model size: 512 × 256 × 256) could achieve a parallel efficiency of approximately 60% with 400K CPU cores using a hybrid, heterogeneous, multiple data programming model.…”

“…Several experimental methods have been developed to analyze the effects of various factors on the CO 2 displacement process, such as wettability, − interfacial tension, ,, pH, , diffusion, − heterogeneity, permeability, and fracture . However, very few studies focus on the effects of injection flow rate and injection pressure difference on the displacement process.…”

“…However, they did not observe any water generation from the preserved cores, and they expressed a lack of clarity about the microflow characteristics of water and oil. Li et al , performed a series of CO 2 displacement experiments with different injection pressure differences, and found that, under immiscible conditions, the total oil recovery had changed from 40.7 to 67.1% as the injection pressure difference increased from 1200 psi (8.27 MPa) to 1800 psi (12.41 MPa). All these studies suggest that the injection flow rate and pressure difference are two important controlling factors in a CO 2 displacement system.…”

Zynq SoC based acceleration of the lattice Boltzmann method

“…The HARVEY code successfully addresses key challenges of image‐based hemodynamics on supercomputers, such as limited memory capacity and bandwidth, flexible load balancing, and scalability. In summary, most of previous HPC implementations mainly provided solutions to exploit cache‐based approach to reuse spatial data for accelerating the LBM computing,() however, those solutions are not optimized for low‐cost and particularly memory limited embedded platforms.…”