The tao of parallelism in algorithms

Abstract

For more than thirty years, the parallel programming community has used the dependence graph as the main abstraction for reasoning about and exploiting parallelism in "regular" algorithms that use dense arrays, such as finite-differences and FFTs. In this paper, we argue that the dependence graph is not a suitable abstraction for algorithms in new application areas like machine learning and network analysis in which the key data structures are "irregular" data structures like graphs, trees, and sets.To address the need for better abstractions, we introduce a datacentric formulation of algorithms called the operator formulation in which an algorithm is expressed in terms of its action on data structures. This formulation is the basis for a structural analysis of algorithms that we call tao-analysis. Tao-analysis can be viewed as an abstraction of algorithms that distills out algorithmic properties important for parallelization. It reveals that a generalized form of data-parallelism called amorphous data-parallelism is ubiquitous in algorithms, and that, depending on the tao-structure of the algorithm, this parallelism may be exploited by compile-time, inspector-executor or optimistic parallelization, thereby unifying these seemingly unrelated parallelization techniques. Regular algorithms emerge as a special case of irregular algorithms, and many application-specific optimization techniques can be generalized to a broader context. These results suggest that the operator formulation and taoanalysis of algorithms can be the foundation of a systematic approach to parallel programming.

Parallel FPGA Routing based on the Operator Formulation

“…We modified all of the data structures used by the router to be thread-safe and compatible with Galois. We rewrote the router to be compliant with the operator formalism [14], which is a central requirement of Galois. We implemented the RRG using Galois' graph model, and the STM-based non-blocking PQ as discussed in Section 4.2.…”

“…It is an irregular algorithm, in the sense that it operates on a sparse graph, typically implemented using a linked data structure. Irregular algorithms do exhibit significant parallelism, but are difficult to parallelize statically [14] because the amount of parallelism depends on the content of the data structure (e.g., graph topology) as well as the operations performed on the elements of the data structure at runtime.…”

“…This paper describes a parallel PathFinder implementation using the open source Galois framework [12,14]. Galois' programming model, compiler, and runtime synergistically accelerate irregular algorithms that dynamically modify linked-based data structures.…”

Multilevel Task Parallelism Exploitation on Asymmetric Sets of Tasks and When Using Third-Party Tools

“…However, taking advantage of such support for multithreading is not always a straight-ahead decision since it often depends on the target platform, the input data set-a collection of data-, among other details. Moreover, in irregular applications-programs that manipulate pointerbased data structures, such as graphs and trees-the amount of parallelism is not predictable at compile time-it is very input dependent [22]-and may be subject of considerable variations at runtime [23]. Unfortunately, neither traditional programming models nor compilers are sufficient to leverage the complexity of exploiting all levels of parallelism in such applications [24][25] [22].…”

Processor Allocation for Optimistic Parallelization of Irregular Programs

“…Algorithms are formulated as iterative computations on work-sets, and each iteration is identified as a quantum of work (task) that can potentially be executed in parallel with other iterations. The Galois project [18] has shown that algorithms formulated in this way can be parallelized automatically using optimistic parallelization): iterations are executed speculatively in parallel and, when an iteration conflicts with concurrently executing iterations, it is rolled-back. Algorithms that have been successfully parallelized in this manner include Survey propagation [5], Boruvka's algorithm [6], Delauney triangulation and refinement [12], and Agglomerative clustering [21].…”