Analysis of system overhead on parallel computers

Abstract

Ever-increasing demand for computing capability is driving the construction of ever-larger computer clusters, typically comprising commodity compute nodes, ranging in size up to thousands of processors, with each node hosting an instance of the operating system (OS). Recent studies El, 41 have shown that even minimal intrusion by the OS on user applications, e.g. a slowdown of user processes of less than 1.0% on each OS instance, can result in a dramatic performance degradation-50% or more-when the user applications are executed on thousands of processors.The contribution of this paper is the explication, and demonstration by way of a case study, of a methodology for analyzing and evaluating the impact of the system (all software and hardware other than user applications) activity on application performance. Our methodology has three major components: 1) a set of simple benchmarks to quickly measure and identdy the impact of intrusive system events; 2) a kernel-level profiling tool OprofiZe to characterize all relevant events and their sources; and, 3) a kernel module that provides timing information for in-depth modeling of the frequency and duration of each relevant event and determines which sources have the greatest impact on performance (and are therefore the most important to eliminate).The paper provides a collection of experimental results conducted on a state-of-the-art dual AMD Opteron duster running GNU/Linux 2.6.5. While our work has been performed on this specific OS, we argue that our contribution readily generalizes to other open source and commercial operating systems.

The Impact of Noise on the Scaling of Collectives: A Theoretical Approach

“…It is well known that many parallel applications do not scale well on large high-performance computing systems [1][2][3]. The per-node performance degradation is more pronounced in systems with more than 1K nodes, running a multi-tasking operating system such as Unix.…”

“…It is increasingly becoming evident that one of the main causes of performance degradation is the noise in the system; in the form of daemons and interrupts, see [1,2].…”

Impact of Noise on Scaling of Collectives: An Empirical Evaluation

“…The Parallel Benchmark (PB) 3 aims to capture the compute-barrier sequence of Figure 1. The kernel of PB is shown in Algorithm-1.…”

“…Scaling of parallel applications on large high-performance computing systems is a well known problem [2][3][4]. Prevalence of large clusters, that uses processors in order of thousands, makes it challenging to guarantee consistent and sustained high performance.…”

“…To overcome variabilities in cluster performance and provide generic methods for tuning clusters for sustained high performance, it is essential to understand theoretically, as well as using empirical data, the behavior of production mode clusters. A known source of performance degradation in large clusters is the noise in the system in the form of daemons and interrupts [2,3]. Impact of OS interference in the form of interrupts and daemons can even cause an order of magnitude performance degradation in certain operations [2,5].…”