Parallel programming

Abstract

The computer industry has a problem. As Moore's law marches on, it will be exploited to double cores, not frequencies. But all those cores, growing to 8, 16 and beyond over the next several years, are of little value without parallel software. Where will this come from? With few exceptions, only graduate students and other strange people write parallel software. Even for numerically intensive applications, where parallel algorithms are well understood, professional software engineers almost never write parallel software. Somehow we need to (1) design many core systems programmers can actually use and (2) provide programmers with parallel programming environments that work. The good news is we have 25+ years of history in the HPC space to guide us. The bad news is that few people are paying attention to this experience. This talk looks at the history of parallel computing to develop a set of anecdotal rules to follow as we create manycore systems and their programming environments. A common theme is that just about every mistake we could make has already been made by someone. So rather than reinvent these mistakes, let's learn from the past and "do it right this time".

Fastflow: High‐Level and Efficient Streaming on Multicore

“…Some of these skeleton frameworks explicitly include stream parallelism as a major source of concurrency exploitation [36,2,34,19]: rather than allowing programmers to connect stages into arbitrary graphs, basic forms of stream parallelism are provided to the programmer in high-level constructs such as pipeline (modeling computations in stages), farm (modeling parallel computation of independent data tasks), and loop (supporting generation of cycles in a stream graph and typically used in combination with a farm body to model Divide&Conquer computations). More recently, approaches such as those followed in algorithmic skeletons but based on parallel design patterns have been claimed to be suitable to support multi-and many-core programming [8,23]. Differences between algorithmic skeletons and parallel design patterns lie mainly in the motivations leading to these two apparently distinct concepts and in the research environments where they were developed: the parallel programming community for algorithmic skeletons and the software engineering community for parallel design patterns.…”

“…FastFlow high-level patterns appear in various algorithmic skeleton frameworks, including Skandium [22], Muesli [10] and Muskel [3]. The parallel design patterns presented in [23] also include equivalents of the FastFlow high-level patterns. These programming frameworks, however, do not specifically address stream programming and so FastFlow outdoes them in terms of efficiency.…”

HPC‐GAP: engineering a 21st‐century high‐performance computer algebra system

“…Parallel pattern that abstracts and re-uses common parallelism idioms is an increasingly important technology [14]. Algorithmic skeletons [15] provide specific implementations of patterns in the form of functions that have effective parallel implementations and that are instantiated with concrete parameters to specify functionality and behaviour.…”

Parallel Haskell implementations of the N‐body problem

“…Therefore, a general trend is to find higher levels of abstractions that can be used to simplify the challenging process of parallelising a complex application. The most successful development in this direction is the use of patterns of parallel computation [38], so called algorithmic skeletons [11], and to instantiate these patterns thereby getting parallelism "for free." The most prominent pattern using this approach is Google's MapReduce pattern [14].…”