The tao of parallelism in algorithms

Abstract

For more than thirty years, the parallel programming community has used the dependence graph as the main abstraction for reasoning about and exploiting parallelism in "regular" algorithms that use dense arrays, such as finite-differences and FFTs. In this paper, we argue that the dependence graph is not a suitable abstraction for algorithms in new application areas like machine learning and network analysis in which the key data structures are "irregular" data structures like graphs, trees, and sets.To address the need for better abstractions, we introduce a datacentric formulation of algorithms called the operator formulation in which an algorithm is expressed in terms of its action on data structures. This formulation is the basis for a structural analysis of algorithms that we call tao-analysis. Tao-analysis can be viewed as an abstraction of algorithms that distills out algorithmic properties important for parallelization. It reveals that a generalized form of data-parallelism called amorphous data-parallelism is ubiquitous in algorithms, and that, depending on the tao-structure of the algorithm, this parallelism may be exploited by compile-time, inspector-executor or optimistic parallelization, thereby unifying these seemingly unrelated parallelization techniques. Regular algorithms emerge as a special case of irregular algorithms, and many application-specific optimization techniques can be generalized to a broader context. These results suggest that the operator formulation and taoanalysis of algorithms can be the foundation of a systematic approach to parallel programming.

A scalable architecture for ordered parallelism

“…Programs leverage Swarm's execution model to convey new work to hardware as soon as it is discovered rather than in the order it needs to run, exposing a large amount of parallelism. As a result, Swarm achieves order-of-magnitude speedups on ordered irregular programs, which are key in emerging domains such as graph analytics, data mining, and in-memory databases [34,55,71]. Swarm hardware could also support thread-level speculation and transactional execution with minimal changes.…”

“…Applications with ordered irregular parallelism have three main characteristics [33,55]. First, they consist of tasks that must follow a total or partial order.…”

“…First, they are common in graph analytics, especially in search problems [33,55]. Second, they are important in simulating systems whose state evolves over time, such as circuits [47], computers [12,59], networks [37,72], healthcare systems [39], and systems of partial differential equations [32,44].…”

“…We focus on ordered irregular parallelism [55], which is often abundant but hard to exploit. Programs with ordered irregular parallelism have three key features.…”

High‐performance computing: to boldly go where no human has gone before

“…This classification contains twelve categories such as graph algorithms, sparse or dense matrix, dynamic programming and so on. In [130], a similar classification was given with more detailed features. In both papers, the idea is to provide efficient libraries (or patterns) to solve each elementary problem on specific computing system.…”