Application and architectural bottlenecks in large scale distributed shared memory machines

Abstract

Many of the programming challenges encountered in small to moderate-scale hardware cache-coherent shared memory machines have been extensively studied. While work remains to be done, the basic techniques needed to efficiently program such machines have been well explored. Recently, a number of researchers have presented architectural techniques for scaling a cache coherent shared address space to much larger processor counts. In this paper, we examine the extent to which applications can achieve reasonable performance on such large-scale, cache-coherent, distributed shared address space machines, by determining the problems sizes needed to achieve a reasonable level of efficiency. We also look at how much programming effort and optimization is needed to achieve high efficiency, beyond that needed at small processor counts. For each application, we discuss the main architectural bottlenecks that prevent smaller problem sizes or less optimized programs from achieving good efficiency. Our results show that while there are some applications that either do not scale or must be heavily optimized to do so, for most of the applications we studied it is not necessary to heavily modify the code or restructure algorithms to scale well upto several hundred processors, once the basic techniques for load balancing and data locality are used that are needed for small-scale systems as well. Programs written with some care perform well without substantially compromising the ease of programming advantage of a shared address space, and the problem sizes required to achieve good performance are surprisingly small. It is important to be careful about how data structures and layouts interact with system granularities, but these optimization are usually needed for moderate-scale machines as well.

Is Data Distribution Necessary in OpenMP?

“…Although one could argue that the overhead of managing parallelism is a problem of the shared memory programming paradigm in general, it has been shown that programs parallelized for shared memory architectures can achieve satisfactory scaling up to a few hundreds of processors [3,4]. This is possible with reasonable scaling of the problem size to increase the granularity of threads and reduce the frequency of synchronization.…”

“…The runtime system records the memory reference trace of the parallel program after the execution of the first iteration. This trace indicates accurately which processor accesses each page more frequently, while the structure of the program ensures that the same reference trace will be repeated throughout the execution of the program, unless the operating system intervenes and preempts or migrates threads 3 . The trace of the first iteration can be used to migrate each page to the node that will minimize the maximum latency due to remote memory accesses to this page, by applying a competitive page migration criterion after the execution of the first iteration.…”

“…and ours resemble in the sense that non-optimal page placement schemes perform often quite close to first-touch under certain architectural circumstances. Some quantitative assessment of page placement schemes appeared also in papers that evaluated the performance of the SPLASH-2 benchmarks on ccNUMA multiprocessors [3,4], however these studies focused on analyzing the locality characteristics of the specific programs.…”

Fallstudie: Parallele Realisierung geophysikalischer Basisalgorithmen in Java

“…Dabei sind sowohl Sockets als auch RMIüberaus komplex, bringen erhebliche zusätzliche Implementierungsarbeit mit sich und erschweren das Verständnis sowie die Wartung der Programme. • Obwohl der Quellcode von MPI-Programmen mit statisch festgelegten Kommunikationsinstruktionen und Thread-Verteilung oft flexibel auf den verschiedenen Architekturen einsetzbar ist, verursacht Portieren häufig erhebliche Leistungseinbußen [5,8]. Darüberhinaus ist es schwierig, Programme mit expliziter Kommunikation an eine sichändernde Anzahl von Knoten, unterschiedliche Architekturen oder verschiedene Netzwerktopologien anzupassen.…”

Global reactive congestion control in multicomputer networks

“…Congestion can be a source of performance degradation in multicomputer networks [3,9]. Congestion arises when the nodes collectively generate traffic which exceeds the network total capacity.…”