A Tight I/O Lower Bound for Matrix Multiplication

Abstract

A tight lower bound for required I/O when computing an ordinary matrix-matrix multiplication on a processor with two layers of memory is established. Prior work obtained weaker lower bounds by reasoning about the number of segments needed to perform C := AB, for distinct matrices A, B, and C, where each segment is a series of operations involving M reads and writes to and from fast memory, and M is the size of fast memory. A lower bound on the number of segments was then determined by obtaining an upper bound on the number of elementary multiplications performed per segment. is paper follows the same high level approach, but improves the lower bound by (1) transforming algorithms for MMM so that they perform all computation via fused multiply-add instructions (FMAs) and using this to reason about only the cost associated with reading the matrices, and (2) decoupling the per-segment I/O cost from the size of fast memory. For n × n matrices, the lower bound's leading-order term is 2n 3 / √ M. A theoretical algorithm whose leading terms a ains this is introduced. To what extent the state-of-the-art Goto's Algorithm a ains the lower bound is discussed.

Automated Derivation of Parametric Data Movement Lower Bounds for Affine Programs

“…Smith et al [Smith et al 2019] introduced a generalization of this argument, leading to tighter bounds in many cases. The idea is to decompose the execution into segments with T loads.…”

“…3.3 on a CDAG G = (V , E), using its compact representation as a DFG. We also present, in 5.1.1, a generalization of one of the techniques introduced in [Dongarra et al 2008;Lowery and Langou 2014;Smith et al 2019; Smith and van de Geijn 2017] that these authors used to derive a tighter lower bound for matrix multiplication.…”

The MOMMS Family of Matrix Multiplication Algorithms

“…Smith et al [26] starts with a simple model of memory with two layers of memory: a small, fast memory with capacity of M elements and a large, slow memory with unlimited capacity. It shows that any algorithm for ordinary MMM 2 must read at least 2mnk/ √ M − 2M elements from slow memory and additionally write at least mn −M elements to slow memory.…”

“…In [26], it is shown that three algorithms, named Resident A, Resident B, and Resident C, a ain the lower bound on the number of reads from slow memory 3 . Additionally, Resident C a ains the lower bound on the number of writes to slow memory 3 .…”

“…is paper first reviews recent theoretical results by Smith et al [26] that establish an (essentially) tight lower bound on the memory traffic incurred by a MMM under a simple model. It then makes a number of new contributions:…”