Communication-Optimal Tilings for Projective Nested Loops with Arbitrary Bounds

Dinh, Grace; Demmel, James

doi:10.1145/3350755.3400275

Cited by 7 publications

(5 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1) Utilization-Driven Objective: High on-chip buffer utilization improves data-reuse opportunity. As demonstrated in the prior work [25], communication lower bounds can be achieved when the tiling block size is optimized for buffer utilization in a system with one-level cache. In this work, we formulate a utilization objective that aims to maximize the buffer utilization of all tensors, so the overall communication is minimized.…”

Section: Objective Functionsmentioning

confidence: 89%

CoSA: Scheduling by Constrained Optimization for Spatial Accelerators

Huang¹,

Kang²,

Dinh³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recent advances in Deep Neural Networks (DNNs) have led to active development of specialized DNN accelerators, many of which feature a large number of processing elements laid out spatially, together with a multi-level memory hierarchy and flexible interconnect. While DNN accelerators can take advantage of data reuse and achieve high peak throughput, they also expose a large number of runtime parameters to the programmers who need to explicitly manage how computation is scheduled both spatially and temporally. In fact, different scheduling choices can lead to wide variations in performance and efficiency, motivating the need for a fast and efficient search strategy to navigate the vast scheduling space.To address this challenge, we present CoSA, a constrainedoptimization-based approach for scheduling DNN accelerators. As opposed to existing approaches that either rely on designers' heuristics or iterative methods to navigate the search space, CoSA expresses scheduling decisions as a constrained-optimization problem that can be deterministically solved using mathematical optimization techniques. Specifically, CoSA leverages the regularities in DNN operators and hardware to formulate the DNN scheduling space into a mixed-integer programming (MIP) problem with algorithmic and architectural constraints, which can be solved to automatically generate a highly efficient schedule in one shot. We demonstrate that CoSA-generated schedules significantly outperform state-of-the-art approaches by a geometric mean of up to 2.5× across a wide range of DNN networks while improving the time-to-solution by 90×.

show abstract

Section: Objective Functionsmentioning

confidence: 89%

CoSA: Scheduling by Constrained Optimization for Spatial Accelerators

Huang¹,

Kang²,

Dinh³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…An input program is a collection of statements enclosed in loop nests, each of the following form (we use the loop nest notation introduced by Dinh and Demmel [23]):…”

Section: Input Programsmentioning

confidence: 99%

“…Pebbling [13,26,37,45,56] Projection-based [8,15,20,21,23,51] Problem specific [1,9,17,48,66] Scope General cDAGs Programs Geometric structure of iteration space Individually tailored for given problem…”

Section: Related Workmentioning

confidence: 99%

On the parallel I/O optimality of linear algebra kernels

Kwasniewski

Ben-Nun

Ziogas

et al. 2021

Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming

View full text Add to dashboard Cite

Matrix factorizations are among the most important building blocks of scientific computing. However, state-of-the-art libraries are not communication-optimal, underutilizing current parallel architectures. We present novel algorithms for Cholesky and LU factorizations that utilize an asymptotically communication-optimal 2.5D decomposition. We first establish a theoretical framework for deriving parallel I/O lower bounds for linear algebra kernels, and then utilize its insights to derive Cholesky and LU schedules, both communicating 3 /( √ ) elements per processor, where M is the local memory size. The empirical results match our theoretical analysis: our implementations communicate significantly less than Intel MKL, SLATE, and the asymptotically communication-optimal CANDMC and CAPITAL libraries. Our code outperforms these state-of-the-art libraries in almost all tested scenarios, with matrix sizes ranging from 2,048 to 524,288 on up to 512 CPU nodes of the Piz Daint supercomputer, decreasing the time-to-solution by up to three times. Our code is ScaLAPACK-compatible and available as an open-source library.

show abstract

“…Let us review previously used techniques to derive communication lower bounds. Algorithms that can be cast as a nested loop program, which encompass many classical linear algebra routines, have relatively simple data access patterns [1,24]. The matrices for their bilinear algorithm consists of columns with one non-zero.…”

Section: Related Workmentioning

confidence: 99%

“…Therefore, it is imperative to design algorithms that minimize communication. Communication lower bounds provide a theoretical limit and guide the design of algorithms that minimize communication [1].…”

Section: Introductionmentioning

confidence: 99%

Communication lower bounds for nested bilinear algorithms

Zhang

Solomonik

2021

Preprint

View full text Add to dashboard Cite

We develop lower bounds on communication in the memory hierarchy or between processors for nested bilinear algorithms, such as Strassen's algorithm for matrix multiplication. We build on a previous framework that establishes communication lower bounds by use of the rank expansion, or the minimum rank of any fixed size subset of columns of a matrix, for each of the three matrices encoding the bilinear algorithm. This framework provides lower bounds for any way of computing a bilinear algorithm, which encompasses a larger space of algorithms than by fixing a particular dependency graph. Nested bilinear algorithms include fast recursive algorithms for convolution, matrix multiplication, and contraction of tensors with symmetry. Two bilinear algorithms can be nested by taking Kronecker products between their encoding matrices.Our main result is a lower bound on the rank expansion of a matrix constructed by a Kronecker product derived from lower bounds on the rank expansion of the Kronecker product's operands. To prove this bound, we map a subset of columns from a submatrix to a 2D grid, collapse them into a dense grid, expand the grid, and use the size of the expanded grid to bound the number of linearly independent columns of the submatrix. We apply the rank expansion lower bounds to obtain novel communication lower bounds for nested Toom-Cook convolution, Strassen's algorithm, and fast algorithms for partially symmetric contractions.

show abstract

Communication-Optimal Tilings for Projective Nested Loops with Arbitrary Bounds

Cited by 7 publications

References 8 publications

CoSA: Scheduling by Constrained Optimization for Spatial Accelerators

CoSA: Scheduling by Constrained Optimization for Spatial Accelerators

On the parallel I/O optimality of linear algebra kernels

Communication lower bounds for nested bilinear algorithms

Contact Info

Product

Resources

About