On Characterizing the Data Access Complexity of Programs

Elango, Venmugil; Rastello, Fabrice; Pouchet, Louis-Noël; Ramanujam, J.; Sadayappan, P.

doi:10.1145/2775051.2677010

Cited by 9 publications

(13 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At the beginning, we were motivated by the characterization of the parallel data movement complexity and dynamic analysis of the data locality potential [2], [3]. As the latency and energy gap increases among the hierarchical layers of the modern computers, it is crucial to understand the data movement complexity of an algorithm, instead of its time complexity.…”

Section: Introductionmentioning

confidence: 99%

Acyclic Partitioning of Large Directed Acyclic Graphs

Herrmann

Kho

Uçar

et al. 2017

2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)

View full text Add to dashboard Cite

Abstract-Finding a good partition of a computational directed acyclic graph associated with an algorithm can help find an execution pattern improving data locality, conduct an analysis of data movement, and expose parallel steps. The partition is required to be acyclic, i.e., the inter-part edges between the vertices from different parts should preserve an acyclic dependency structure among the parts. In this work, we adopt the multilevel approach with coarsening, initial partitioning, and refinement phases for acyclic partitioning of directed acyclic graphs and develop a direct k-way partitioning scheme. To the best of our knowledge, no such scheme exists in the literature. To ensure the acyclicity of the partition at all times, we propose novel and efficient coarsening and refinement heuristics. The quality of the computed acyclic partitions is assessed by computing the edge cut, the total volume of communication between the parts, and the critical path latencies. We use the solution returned by well-known undirected graph partitioners as a baseline to evaluate our acyclic partitioner, knowing that the space of solution is more restricted in our problem. The experiments are run on large graphs arising from linear algebra applications.

show abstract

Section: Introductionmentioning

confidence: 99%

Acyclic Partitioning of Large Directed Acyclic Graphs

Herrmann

Kho

Uçar

et al. 2017

2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)

View full text Add to dashboard Cite

show abstract

“…Second, a large body of related work is concerned with autoparallelization of imperative code. Such work includes static dependency analyses [Feautrier 1991;Pouchet et al 2011;Lin and Padua 2000] for recognizing and extracting loop-level parallelism, as well as for characterizing/improving memory access patterns [Elango et al 2015]. Other techniques may extract partial parallelism dynamically [Dang et al 2002;, but such techniques have not been evaluated (yet) on GPGPUs.…”

Section: Related Workmentioning

confidence: 99%

FinPar

Andreetta¹,

Bégot²,

Berthold

et al. 2016

ACM Trans. Archit. Code Optim.

View full text Add to dashboard Cite

Commodity many-core hardware is now mainstream, but parallel programming models are still lagging behind in efficiently utilizing the application parallelism. There are (at least) two principal reasons for this. First, real-world programs often take the form of a deeply nested composition of parallel operators, but mapping the available parallelism to the hardware requires a set of transformations that are tedious to do by hand and beyond the capability of the common user. Second, the best optimization strategy, such as what to parallelize and what to efficiently sequentialize, is often sensitive to the input dataset and therefore requires multiple code versions that are optimized differently, which also raises maintainability problems. This article presents three array-based applications from the financial domain that are suitable for GPGPU execution. Common benchmark-design practice has been to provide the same code for the sequential and parallel versions that are optimized for only one class of datasets. In comparison, we document (1) all available parallelism via nested map-reduce functional combinators, in a simple Haskell implementation that closely resembles the original code structure, (2) the invariants and code transformations that govern the main trade-offs of a data-sensitive optimization space, and (3) report target CPU and multiversion GPGPU code together with an evaluation that demonstrates optimization trade-offs and other difficulties. We believe that this work provides useful insight into the language constructs and compiler infrastructure capable of expressing and optimizing such applications, and we report in-progress work in this direction.

show abstract

“…Analyzing I/O bounds of linear algebra kernels dates back to the seminal work by Hong and Kung [8], who derived the first asymptotic bound for matrix-matrix multiplication (MMM) using the red-blue pebble game abstraction. This method was subsequently extended and used by other works to derive asymptotic [15] and tight [16] bounds for more complex programs. Despite the expressiveness of pebbling, it is prohibitively hard to solve for arbitrary programs, as it is PSPACE-complete in the general case [17].…”

Section: Introductionmentioning

confidence: 99%

Pebbles, Graphs, and a Pinch of Combinatorics: Towards Tight I/O Lower Bounds for Statically Analyzable Programs

Kwasniewski,

Ben-Nun,

Gianinazzi

et al. 2021

Preprint

View full text Add to dashboard Cite

Determining I/O lower bounds is a crucial step in obtaining communication-efficient parallel algorithms, both across the memory hierarchy and between processors. Current approaches either study specific algorithms individually, disallow programmatic motifs such as recomputation, or produce asymptotic bounds that exclude important constants. We propose a novel approach for obtaining precise I/O lower bounds on a general class of programs, which we call Simple Overlap Access Programs (SOAP). SOAP analysis covers a wide variety of algorithms, from ubiquitous computational kernels to full scientific computing applications. Using the red-blue pebble game and combinatorial methods, we are able to bound the I/O of the SOAP-induced Computational Directed Acyclic Graph (CDAG), taking into account multiple statements, input/output reuse, and optimal tiling. To deal with programs that are outside of our representation (e.g., non-injective access functions), we describe methods to approximate them with SOAP. To demonstrate our method, we analyze 38 different applications, including kernels from the Polybench benchmark suite, deep learning operators, and -for the first time -applications in unstructured physics simulations, numerical weather prediction stencil compositions, and full deep neural networks. We derive tight I/O bounds for several linear algebra kernels, such as Cholesky decomposition, improving the existing reported bounds by a factor of two. For stencil applications, we improve the existing bounds by a factor of up to 14. We implement our method as an open-source tool, which can derive lower bounds directly from provided C code.

show abstract

On Characterizing the Data Access Complexity of Programs

Cited by 9 publications

References 38 publications

Acyclic Partitioning of Large Directed Acyclic Graphs

Acyclic Partitioning of Large Directed Acyclic Graphs

FinPar

Pebbles, Graphs, and a Pinch of Combinatorics: Towards Tight I/O Lower Bounds for Statically Analyzable Programs

Contact Info

Product

Resources

About