An efficient tensor transpose algorithm for multicore CPU, Intel Xeon Phi, and NVidia Tesla GPU

Abstract

a b s t r a c tAn efficient parallel tensor transpose algorithm is suggested for shared-memory computing units, namely, multicore CPU, Intel Xeon Phi, and NVidia GPU. The algorithm operates on dense tensors (multidimensional arrays) and is based on the optimization of cache utilization on x86 CPU and the use of shared memory on NVidia GPU. From the applied side, the ultimate goal is to minimize the overhead encountered in the transformation of tensor contractions into matrix multiplications in computer implementations of advanced methods of quantum many-body theory (e.g., in electronic structure theory and nuclear physics). A particular accent is made on higher-dimensional tensors that typically appear in the so-called multireference correlated methods of electronic structure theory. Depending on tensor dimensionality, the presented optimized algorithms can achieve an order of magnitude speedup on x86 CPUs and 2-3 times speedup on NVidia Tesla K20X GPU with respect to the naïve scattering algorithm (no memory access optimization). The tensor transpose routines developed in this work have been incorporated into a general-purpose tensor algebra library (TAL-SH).

Coupled‐cluster singles, doubles and perturbative triples with density fitting approximation for massively parallel heterogeneous platforms

“…This effort was later extended to the evaluation of the terms in renormalized CCSD(T) and multireference CCSD(T) . More recently Lyakh developed a standalone general‐purpose tensor algebra library (TAL_SH) which supports basic tensor algebra on multicore CPU, many‐core, and NVIDIA GPU‐containing shared‐memory computers; although TAL_SH includes custom CUDA kernels for tensor operations, vendor provided BLAS is recommended to reach high performance. Yet another effort to produce optimized device kernels for tensor algebra was described by Kim et al who considered specifically the tensor contractions that appear in the (T) energy.…”

Establishing the quantum supremacy frontier with a 281 Pflop/s simulation

“…The tensor contraction operation in TAL-SH is implemented by the general Transpose-Transpose-GEMM-Transpose (TTGT) algorithm, with an optimized GPU tensor transpose operation [57] (see also Ref. [58]).…”

HPTT: a high-performance tensor transposition C++ library

“…Lyakh et al [12] designed a generic multidimensional transpose algorithm and evaluated it across different architectures (e.g., Intel Xeon, Intel Xeon Phi, AMD and NVIDIA K20X). Despite the fact that their algorithm outperforms a naive baseline implementation, the results suggest that there still exists a noticeable performance gap to the bandwidth attained by a direct copy.…”