Modelling the Earth's geomagnetic environment on Cray machines using PETSc and SLEPc

Abstract

The British Geological Survey's global geomagnetic model, Model of the Earth's Magnetic Environment (MEME), is an important tool for calculating the strength and direction of the Earth's magnetic field, which is continually in flux. While the ability to collect data from ground-based observation sites and satellites has grown rapidly, the memory bound nature of the original code has proved a significant limitation on the size of the modelling problem required. In this paper, we describe work done replacing the bespoke, sequential, eigensolver with that of the PETSc/SLEPc package for solving the system of normal equations. Adopting PETSc/SLEPc also required fundamental changes in how we built and distributed the data structures, and as such, we describe an approach for building symmetric matrices that provides good load balance and avoids the need for close coordination between the processes or replication of work. We also study the memory bound nature of the code from an irregular memory accesses perspective and combine detailed profiling with software cache prefetching to significantly optimise this. Performance and scaling characteristics are explored on ARCHER, a Cray XC30, where we achieved a speed up for the solver of 294 times by replacing the model's bespoke approach with SLEPc.

Foreword to the Special Issue of the Cray User Group

“…Brown et al describe a significant improvement of the British Geological Survey's global geomagnetic model code, involving a replacement of its custom eigensolver with the scalable library for eigenvalue problem computations (SLEPc) and the portable, extensible toolkit for scientific computations ( PETSc), along with code modernization and in‐depth performance analysis related to memory accesses. The results show a performance improvement of up to 300 times over the existing implementation.…”

The BGS candidate models for IGRF-13 with a retrospective analysis of IGRF-12 secular variation forecasts

“…Field modelling within BGS has matured since the release of the IGRF-12 candidate models of Hamilton et al (2015), with higher temporal and spatial resolution of the secular variation using order 6 B-splines, and the improvement of core flow advection using steady flow and steady flow acceleration to predict the secular variation. In addition, the underlying code base has been modernised and modularised, allowing massively parallel inverse solvers such as PETSc (Balay et al 1997) and SLEPc (Hernandez et al 2005) to be used (see Brown et al 2020).…”