Fast and flexible implementations of Wilson, Brillouin and Susskind fermions in lattice QCD

Abstract

A modern Fortran implementation of three Dirac operators (Wilson, Brillouin, Susskind) in lattice QCD is presented, based on OpenMP shared-memory parallelization and SIMD pragmas. The main idea is to apply a Dirac operator to N v vectors simultaneously, to ease the memory bandwidth bottleneck. All index computations are left to the compiler and maximum weight is given to portability and flexibility. The lattice volume, N x N y N z N t , the number of colors, N c , and the number of right-hand sides, N v , are parameters defined at compile time. Several memory layout options are compared. The code performs well on modern many-core architectures (480 Gflop/s, 880 Gflop/s, and 780 Gflop/s with N v = 12 for the three operators in single precision on a 72-core KNL processor, a 2×24-core Skylake node yields similar results). Explicit run-time tests with CG/BiCGstab inverters confirm that the memory layout is relevant for the KNL, but less so for the Skylake architecture. The ancillary code distribution contains all routines, including the single, double, and mixed precision Krylov space solvers, to render it self-contained and ready-to-use.1 Here "frequently" means O(10 5 ) times, "large" implies a n × n matrix with n = 402 653 184 for a Wilson fermion on a 64 3 × 128 lattice, and depending on the quark mass the condition number of D † D is often in the range 10 6 . . . 10 8 . The factor 10 5 reflects the production of an ensemble of 1000 gauge configurations, separated by ten τ = 1 HMC trajectories, assuming that each of these requires O(10) inversions.