A case study in mechanically deriving dense linear algebra code

Abstract

Design by Transformation (DxT) is a top-down approach to mechanically derive high-performance algorithms for dense linear algebra. We use DxT to derive the implementation of a representative matrix operation, two-sided Trmm. We start with a knowledge base of transformations that were encoded for a simpler set of operations, the level-3 BLAS, and add only a few transformations to accommodate the more complex two-sided Trmm. These additions explode the search space of our prototype system, DxTer, requiring the novel techniques defined in this paper to eliminate large segments of the search space that contain suboptimal algorithms. Performance results for the mechanically optimized implementations on 8192 cores of a BlueGene/P architecture are given.

Interfaces are key

“…Well designed interfaces enable a variety of algorithms to be implemented quickly for prototyping and deployment. Automation takes this a step forward by automating the implementation process and even the exploration of algorithmic options [8]. Such automation is not possible without good interface design.…”

Understanding performance stairs

“…For distributed-memory DLA, first-order cost estimates are sufficient [17,18,19] to enable an expert to judge trade offs between the cost of communicating data over a network and increasing parallelism that is enabled by that communication. Just as an expert estimates efficiency when manually coding, DxTer does so automatically by summing the estimated runtime of all nodes on a graph [19].…”

“…We have automated the exploration of these spaces (by generating all implementations using a methodical process) and we evaluate the efficiency of each implementation via cost estimation. 1 This is how we find the best-performing algorithm that experts would intuitively select [17,18,19]. In all tests, generated code is the same or better than experts' hand-produced implementations.…”

“…We begin with a brief overview of how we generate the space of implementations for a given operation in the domain. Our approach is called Design by Transformation (DxT) -more details are given in [9,17,18,19,27]. …”

Reliable Generation of High-Performance Matrix Algebra

“…The DxT project [Marker et al 2013] uses a cost model based on operation count and communication costs to estimate the performance of many possible implementations of distributed-memory dense linear algebra, by composing each algorithm mostly out of Level 3 BLAS subroutines. They use a similar style of search heuristics to narrow the space, focusing on transformations likely to be helpful.…”