Evaluation of parallel direct sparse linear solvers in electromagnetic geophysical problems

Abstract

High performance computing is absolutely necessary for large-scale geophysical simulations. In order to obtain a realistic image of a geologically complex area, industrial surveys collect vast amounts of data making the computational cost extremely high for the subsequent simulations. A major computational bottleneck of modeling and inversion algorithms is solving the large sparse systems of linear ill-conditioned equations in complex domains with multiple right hand sides. Recently, parallel direct solvers have been successfully applied to multi-source seismic and electromagnetic problems. These methods are robust and exhibit good performance, but often require large amounts of memory and have limited scalability. In this paper, we evaluate modern direct solvers on large-scale modeling examples that previously were considered unachievable with these methods. Performance and scalability tests utilizing up to 65,536 cores on the Blue Waters supercomputer clearly illustrate the robustness, efficiency and competitiveness of direct solvers compared to iterative techniques. Wide use of direct methods utilizing modern parallel architectures will allow modeling tools to accurately support multi-source surveys and 3D data acquisition geometries, thus promoting a more efficient use of the electromagnetic methods in geophysics.The authors would like to thank the MUMPS and PARDISO developers for providing free academic licenses and Dr. Anshul Gupta for access to his solver library which is not publicly available. We also would like to thank the Private Sector Program and the Blue Waters sustained-petascale computing project at the\ud National Center for Supercomputing Applications (NCSA), which is supported by the National Science Foundation (awards OCI-\ud 0725070 and ACI-1238993) and the state of Illinois. The shared-memory tests were performed on the MareNostrum supercomputer of the Barcelona Supercomputer Center. The first author\ud acknowledges funding from the Repsol-BSC Research Center through the AURORA project and support from the RISE Horizon 2020 European Project GEAGAM (644602). The authors wish to thank Jef Caers and two anonymous reviewers for their valuable\ud comments that significantly helped to improve this paper.Peer ReviewedPostprint (author's final draft

Parallel Refined Isogeometric Analysis in 3D

“…In addition to the above theoretical considerations, we know that different direct solver implementations exhibit distinct scalability properties, as shown in [34]. Therein, the authors consider specific finite differences and finite element systems and demonstrate via numerical experimentation that for those systems, the parallel scalability of solvers Watson Sparse Matrix Package (WSMP) [22] and MUMPS exhibit large discrepancies.…”

Large-scale 3-D EM modelling with a Block Low-Rank multifrontal direct solver

“…The numbers represent the approximate number of unknowns, in millions, in the linear systems associated to each matrix; for example, the linear system corresponding to matrix S21 had about 21 million unknowns. So far, for 3-D geophysical EM problems, the largest reported complex-valued linear system that has been solved with a direct solver had 7.8 million unknowns (Puzyrev et al 2016).…”

“…Therefore the application of direct solvers to 3-D problems has traditionally been considered computationally too demanding. However, recent advances in sparse matrix-factorization packages, for example, MUMPS (Amestoy et al 2001(Amestoy et al , 2006, PARDISO (Schenk & Gärtner 2004), SuperLU (Li & Demmel 2003), UMFPACK (Davis 2004) and WSMP (Gupta & Avron 2000), along with the availability of modern parallel computing environments, have created the necessary conditions to attract interest in direct solvers in the case of 3-D EM problems of moderate size, see for example, Blome et al (2009), Streich (2009), da Silva et al (2012, Puzyrev et al (2016) and Jaysaval et al (2014).…”

A multi-domain decomposition-based Fourier finite element method for the simulation of 3D marine CSEM measurements

“…By noticing that basis functions Ψ kq , k = 1, ..., K q , q = 1, .., N s have support over Ω q 2 and the subdomains Ω i 2 , i = 1, ..., N s are non-connected, (25) can be expressed as:…”

“…Typically, the number of sources is in the order of tens to hundreds for land CSEM studies, while it reaches thousands in modern large-scale marine CSEM surveys [21]. Puzyrev et al [25] evaluated modern direct solvers on large-scale geophysical simulations that previously were considered unachievable with these methods.…”