A performance analysis of local synchronization

Abstract

Synchronization is often necessary in parallel computing, but it can create delays whenever the receiving processor is idle, waiting for the information to arrive. This is especially true for barrier, or global, synchronization, in which every processor must synchronize with every other processor. Nonetheless, barriers are the only form of synchronization explicitly supplied in MPI and OpenMP.Many applications do not actually require global synchronization; local synchronization, in which a processor synchronizes only with those processors from which it has an incoming edge in some directed graph, is often adequate. However, the behavior of a system under local synchronization is more difficult to analyze, since processors do not all start tasks at the same time.In this paper, we show that if the synchronization graph is a directed cycle and the task times are geometrically distributed with p = 0.5, the time it takes for a processor to complete a task, including synchronization time, approaches an exact limit of 2 + √ 2 as the number of processors in the cycle approaches infinity. Under global synchronization, however, the time is unbounded, increasing logarithmically with the number of processors. Similar results also apply for p = 0.5.We give a new proof of the constant upper bounds that apply when tasks are normally distributed and the synchronization graph is any graph of bounded degree. We also prove that for some power-law distributions on the tasks, there is no constant upper bound as the number of processors increases, even for the directed cycle. Finally, we show that constant upper bounds apply for some cases of a different synchronization model in which a processor waits for only a subset of its neighbors.

On bottleneck analysis in stochastic stream processing

“…Closest related work by Rajsbaum and Sidi [1994] and, more recently, by Lipman and Stout [2006], studied the impact of random processing times and transmission delays on the average number of computational steps executed by a processor in the network per unit time when attempting to synchronize over a distributed network. Our work differs in two notable ways.…”

On the costs and benefits of stochasticity in stream processing

“…We seek to characterize the mean end-to-end latency (MEEL), the variance, and whenever possible provide a complete analytical description of EEL via its probability distribution. Closest related work by Rajsbaum and Sidi [25] and, more recently, by Lipman and Stout [20], studied the impact of random processing times and transmission delays on the average number of computational steps executed by a processor in the network per unit time when attempting to synchronize over a distributed network. Our work differs in two notable ways.…”