Strassen's Algorithm for Tensor Contraction

Abstract

Tensor contraction (TC) is an important computational kernel widely used in numerous applications. It is a multi-dimensional generalization of matrix multiplication (GEMM). While Strassen's algorithm for GEMM is well studied in theory and practice, extending it to accelerate TC has not been previously pursued. Thus, we believe this to be the first paper to demonstrate how one can in practice speed up tensor contraction with Strassen's algorithm. By adopting a Block-Scatter-Matrix format, a novel matrix-centric tensor layout, we can conceptually view TC as GEMM for a general stride storage, with an implicit tensor-to-matrix transformation. This insight enables us to tailor a recent state-of-the-art implementation of Strassen's algorithm to TC, avoiding explicit transpositions (permutations) and extra workspace, and reducing the overhead of memory movement that is incurred. Performance benefits are demonstrated with a performance model as well as in practice on modern single core, multicore, and distributed memory parallel architectures, achieving up to 1.3× speedup. The resulting implementations can serve as a drop-in replacement for various applications with significant speedup.

Efficient computation of the second-Born self-energy using tensor-contraction operations

“…We anticipate the speedup may be even more advantageous when the code is executed in parallel, taking full advantage of the optimized underlying numerical libraries. Instead of looping over the basis indices, utilizing efficiently optimized external numerical libraries for the tensor-contraction operations has the further advantage of speeding up the computation if/when future implementations of the the external libraries become and even more efficient [45]. The presented approach is not limited to the 2B self-energy only but could be readily used for other correlation self-energies, such as GW or T-matrix.…”

“…4. This reduced scaling could be related to the optimization of matrix multiplication using Strassen [64] or Coppersmith-Winograd [65] algorithms, and to more advanced methods for tensor contraction algorithms which can scale faster than the naïve looping scheme [45].…”

“…From the resulting structures of the internal summations in the self-energy diagrams, we identify matrix or tensor operations. Instead of simply looping over the basis indices, employing well-established linear algebra libraries for the matrix and tensor operations [43][44][45] may speed up the construction of the self-energy.…”

“…In this paper we propose to transform the summation expressions in the self-energy diagrams using tensorcontraction operations, and to further employ external linear algebra libraries (e.g. lowlevel C or Fortran) taking into account, e.g., memory availability, communication costs, loop fusion and ordering [43][44][45]. (Here we consider tensors simply as multidimensional objects without deeper (differential-)geometric interpretation.)…”

Optimization of Finite-Differencing Kernels for Numerical Relativity Applications

“…Tensor contractions could be further optimized with the use of dedicated libraries, e.g., [21][22][23][24]. As noted in [21,24], however, the performance improvement is neither obvious nor guaranteed because compilers optimization are very effective on explicitly coded loops.…”