Integrated Loop Optimizations for Data Locality Enhancement of Tensor Contraction Expressions

Abstract

A very challenging issue for optimizing compilers is the phase ordering problem: In what order should a collection of compiler optimizations be performed? We address this problem in the context of optimizing a sequence of tensor contractions. The pertinent loop transformations are loop permutation, tiling, and fusion; in addition, the placement of disk I/O statements crucially affects performance. The space of possible combinations is exponentially large. We develop novel pruning strategies whereby a search problem in a larger space is replaced by a large number of searches in a much smaller space, to determine the optimal permutation, fusion, tiling and placement of disk I/O statements. Experimental results show that we obtain an improvement in I/O cost by a factor of up to 2.6 over an equi-tile-size approach. Proceedings of the 2005 ACM/IEEE SC|05 Conference (SC'05) Proceedings of the 2005 ACM/IEEE SC|05 Conference (SC'05) Proceedings of the 2005 ACM/IEEE SC|05 Conference (SC'05)

Practical Loop Transformations for Tensor Contraction Expressions on Multi-level Memory Hierarchies

“…The key decisions to be made are 1) determining the loop structure, comprising of fused and tiled loops, 2) the placement of memory allocation and data movement statements for each of the memory hierarchy levels, and 3) the tile sizes. It has been shown previously that the the space of possible choices is exponentially large [33]. Besides considering additional levels in the memory hierarchy in determining fusion structures, we focus on the prescriptive approaches to choosing candidate loop fusions without requiring expensive optimization procedures.…”

“…from disks to main memory). A detailed description is available from an earlier publication [33]. In a tensor contraction, the indices can be grouped into those that are contracted and the remaining indices in the two input tensors.…”

“…It can be shown that to minimize the data movement costs, the smallest array is retained in memory, with the other two arrays being streamed. This results in a memory cost of [33]:…”

“…Our work is specifically in the context of tensor contractions [33,14], which can be viewed as generalized matrix products. Sequences of tensor contractions arise in several domains, particularly, in ab initio computational models in quantum chemistry.…”

“…Specifically, we consider data movement costs under memory constraints. For systems with only two levels of memory hierarchy, this problem has been addressed before by Sahoo et al [33].…”