Solving DWF Dirac Equation Using Multisplitting Preconditioned Conjugate Gradient

Abstract

We show that using the multisplitting algorithm as a preconditioner for conjugate gradient inversion of the domain wall Dirac operator could effectively reduce the internode communication cost, at the expense of performing more on-node floating point operations. This method could be useful for supercomputers with far more on-node flops than inter-node communication bandwidth.

2+1 Flavor Domain Wall Fermion QCD Lattices: Ensemble Production and (some) Properties

“…This is very little bandwidth for strong scaling the solvers used for DWF QCD to, say, 1024 nodes, since the local volumes implied by a 1024 node job generally require of order one byte of off-node bandwidth per sustained Flop. To increase local (on-node) floating point utilization in the (M)DWF conjugate gradient, we have developed the Multisplitting Preconditioned Conjugate Gradient (MSPCG) [5,6]. The Multisplitting algorithm [7] provides general criteria for detailing how a linear equation solve can be split into submatrix pieces, with each solved separately, and then an update step is done, which spans the submatricies, to redefine the next iteration of the problem.…”

“…which connects fourth nearest-neighbor sites on the lattice (see [5,6]) we originally failed to get convergence. Our initial approach split the underlying four-dimensional Wilson Dirac operator (M 4 ) into submatrices localized on each GPU, i.e.…”

“…Changing so that we were splitting D † PC D PC into submatrices made the Multisplitting algorithm converge, but it converged more slowly than the original CG. However, we were able to use the local submatrix decomposition as a preconditioner to the CG and found that the CG iteration count fell by a factor of 3 for physical light quark masses [5]. (Using the multisplitting algorithm as a preconditioner for the CG, when the submatrix decomposition is the natural one given by the underlying lattice geometry, makes our implementation the same as using the additive Schwarz algorithm as a preconditioner.…”

“…We adjusted the number of poles in the rational approximation needed in the EOFA heat bath, and tightened the stopping condition for the sovers calculating this part of the Hamiltonian, and the occasional outliers have gone away. In particular for trajectories 300 to 730, exp(−∆H) = 0.94 (5), where the error comes from assuming that values of ∆H are decorrelated for each trajectory.…”