Thread-level parallelization and optimization of NWChem for the Intel MIC architecture

Abstract

In the multicore era it was possible to exploit the increase in on-chip parallelism by simply running multiple MPI processes per chip. Unfortunately, manycore processors' greatly increased thread-and data-level parallelism coupled with a reduced memory capacity demand an altogether different approach. In this paper we explore augmenting two NWChem modules, triples correction of the CCSD(T) and Fock matrix construction, with OpenMP in order that they might run efficiently on future manycore architectures. As the next NERSC machine will be a self-hosted Intel MIC (Xeon Phi) based supercomputer, we leverage an existing MIC testbed at NERSC to evaluate our experiments. In order to proxy the fact that future MIC machines will not have a host processor, we run all of our experiments in native mode. We found that while straightforward application of OpenMP to the deep loop nests associated with the tensor contractions of CCSD(T) was sufficient in attaining high performance, significant effort was required to safely and efficiently thread the TEXAS integral package when constructing the Fock matrix. Ultimately, our new MPI+OpenMP hybrid implementations attain up to 65× better performance for the triples part of the CCSD(T) due in large part to the fact that the limited on-card memory limits the existing MPI implementation to a single process per card. Additionally, we obtain up to 1.6× better performance on Fock matrix constructions when compared with the best MPI implementations running multiple processes per card.

An efficient MPI/OpenMP parallelization of the Hartree–Fock–Roothaan method for the first generation of Intel® Xeon Phi™ processor architecture

“…The approach was tested in the FMO and Community Earth System Model (CESM) packages. Shan et al (2014Shan et al ( , 2015 used OpenMP task parallelism to HF SCF and CCSD(T) drivers.…”

Improving efficiency of semi‐direct møller–plesset second‐order perturbation methods through oversubscription on multiple nodes

“…While in some cases, the addition of OpenMP threads improves performance, neither NWChem nor the Global Arrays toolkit are completely thread safe. The use of OpenMP with NWChem CCSD(T) calculations was shown to improve performance, but extensive changes were required in every routine accessed during the calculations, and even variables within nested loops had to be updated.…”

A Guide for Achieving High Performance with Very Small Matrices on GPU: A Case Study of Batched LU and Cholesky Factorizations

“…Moreover, in [34], the authors discussed the optimization of NWChem for Intel's MIC architecture and highlighted the need for tensor computations of about 200-2,000 matrices from 10 × 10 to 40 × 40 in size. In his dissertation, David Ozog discussed NWChem's Tensor Contraction Engine (TCE) and revealed how strongly it relies on the performance of general matrix-matrix multiplication (GEMM) in the computation of the tensor contraction.…”