Bulk Synchronous Parallelism (BSP) is a parallel programming model that abstracts from low-level program structures in favour of supersteps. A superstep consists of a set of independent local computations, followed by a global communication phase and a barrier synchronisation. Structuring programs in this way enables their costs to be accurately determined from a few simple architectural parameters, namely the permeability of the communication network to uniformly-random traffic and the time to synchronise. Although permutation routing and barrier synchronisations are widely regarded as inherently expensive, this is not the case. As a result, the structure imposed by BSP does not reduce performance, while bringing considerable benefits for application building. This paper answers the most common questions we are asked about BSP and justifies its claim to be a major step forward in parallel programming.
Why Is Another Model Needed?In the 1980s, a large number of different types of parallel architectures were developed. This variety may have been necessary to thoroughly explore the design space but, in retrospect, it had a negative effect on the commercial development of parallel applications software. To achieve acceptable performance, software had to be carefully tailored to the specific architectural properties of each computer, making portability almost impossible. Each new generation of processors appeared in strikingly-different parallel architectural frameworks, forcing performancedriven software developers to redesign their applications from the ground up. Understandably, few were keen to join this process.Today, the number of parallel computation models and languages probably exceeds the number of different architectures with which parallel programmers had to contend ten years ago. Most make it hard to achieve portability, hard to achieve performance, or both. The two largest classes of models are based on message passing, and on shared memory. Those based on message passing are inadequate for three reasons. First, messages require paired actions at the sender and receiver, which it is difficult to ensure are correctly matched. Second, messages blend communication and synchronisation so that sender and receiver must be in appropriately-consistent states when the communication takes place. This is appallingly difficult to ensure in most models, and programs are prone to deadlock as a result. Third, the performance of such programs is impossible to predict because the interaction of large numbers of individual messages in the interconnection mechanism makes the variance in their delivery times large.The argument for shared-memory models is that they are easier to program because they provide the abstraction of a single, shared address space. A whole class of placement decisions are avoided. This is true, but is only half of the issue. When memory is shared, simultaneous access to the same location must be prevented. This requires either PRAM-style discipline by the programmer, or expensive lock mana...
