The impact of network noise at large-scale communication performance

Hoefler, Torsten; Schneider, Timo; Lumsdaine, Andrew

doi:10.1109/ipdps.2009.5161095

Cited by 28 publications

(18 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We attribute this variation to interference with other applications (cf. network noise in [12]). However, using the statistical methods explained in Section III-A, we are able to provide the relative variance for each of the parameters in brackets.…”

Section: Performance On Various Architecturesmentioning

confidence: 99%

Performance Modeling and Comparative Analysis of the MILC Lattice QCD Application su3_rmd

Bauer

Gottlieb

Hoefler

2012

2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (Ccgrid 2012)

Self Cite

View full text Add to dashboard Cite

Abstract-Application performance modeling is an essential part of application and system development as HPC moves into the petascale and prepares for the exascale. However, performance modeling of parallel systems is a difficult task due to natural variations in measurements and noise effects. In this paper, we give a detailed example for a semi-analytical performance-modeling method applied to the ubiquitous HPC application su3 rmd from the lattice Quantum Chromodynamics field on a variety of parallel computing platforms. We apply statistical techniques that are well known in natural sciences to model the variance in the input system. Using a simple analytical model to capture the main characteristics of the code, such as numbers and sizes of passed messages and invocation counts of serial code blocks in conjunction with statistically sound curvefitting methods, we develop an accurate performance model and use it to characterize application performance on various target architectures. Our fitting techniques allow us to characterize the variance of different performance observations on a given system and show the influence of noise from different sources. The techniques we developed can be applied to a wide class of bulk-synchronous applications. With this detailed example, we aim to motivate the scientific computing community to develop and use similar performance models for software development and maintenance.

show abstract

Section: Performance On Various Architecturesmentioning

confidence: 99%

Performance Modeling and Comparative Analysis of the MILC Lattice QCD Application su3_rmd

Bauer

Gottlieb

Hoefler

2012

2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (Ccgrid 2012)

Self Cite

View full text Add to dashboard Cite

show abstract

“…More information about ORCS and the source-code are available in [8] and at http://www.unixer.de/research/orcs/. Based on ORCS simulation results, we analyzed the impact of Network Noise (due to background network congestion) on collective algorithms and applications [3].…”

Section: Activitiesmentioning

confidence: 99%

“…The research completed in this period has improved understanding of when such noise will have a significant impact. Specifically, it was demonstrated that not just noise on the nodes, but also noise on the network between nodes can have a significant impact on computation time [3]. It was also demonstrated that noise patterns matter more than noise intensity: very regular noise can cause less disruption than lighter (on average) but less regular noise [9].…”

mentioning

confidence: 99%

Award ER25844: Minimizing System Noise Effects for Extreme-Scale Scientific Simulation Through Function Delegation

Lumsdaine¹

2012

Self Cite

View full text Add to dashboard Cite

Executive SummaryIn software running on distributed computing clusters, time spent on communication between nodes in the cluster can be a significant portion of the overall computation time; background operating system tasks and other computational "noise" on the nodes of the system can have a significant impact on the amount of time this communication takes, especially on large systems. The research completed in this period has improved understanding of when such noise will have a significant impact. Specifically, it was demonstrated that not just noise on the nodes, but also noise on the network between nodes can have a significant impact on computation time [3]. It was also demonstrated that noise patterns matter more than noise intensity: very regular noise can cause less disruption than lighter (on average) but less regular noise [9]. It was also demonstrated that the effect of noise is more prominent as the speed of the network between nodes is increased [9]. Furthermore, a tracing tool, Netgauge, was improved via our work, and a system simulator, LogGOPSim, was developed; they can be used by application developers to improve performance of their program and by system designers to mitigate the effects of noise by adjusting the noise characteristics of the operating system. Both have been made freely available as open source programs. In the course of developing these tools, we demonstrated weaknesses in existing methodologies for modeling communication [1], and we introduced a more detailed model, LogGOPS, for simulating systems.Not only were the deleterious effects of noise explored but we have also offered solutions. Our studies of simulations of system noise have led to specific recommendations on tuning systems to mitigate noise. We have also improved existing approaches to mitigating noise. "Non-blocking collective communication" avoids the effects of noise by letting communication continue simultaneously with computation (thus being "non-blocking"), so that the delays in communication introduced by noise have a smaller impact on overall computation time. Potentially, noise can be reduced much further by "offloading" communication tasks to a separate processing element than the operating system is using.We have improved our library LibNBC, which provides an implementation of non-blocking collectives, via this work. During this research, our proposal to include non-blocking collectives (which used LibNBC as a reference implementation) in the upcoming MPI-3 standard was accepted. As MPI is a ubiquitous and important standard for communication in parallel computing, this demonstrates a certain acceptance of the practicality and desirability of non-blocking collectives. Now that non-blocking collectives are a part of the standard we can expect to see optimized platform-specific implementations of non-blocking collectives. Also as part of this work we have also developed a language GOAL (Global Operation Assembly Language) that can be used as a starting point for defining languages to express optimized com...

show abstract

“…This is also coupled with the availability of advanced network hardware which is capable for offloading communication operations. Such offloaded collective communication is beneficial to systems that can be prone to OS noise [3], [8]. Finally, applications stand to gain significantly from nonblocking collectives by overlapping computation and communication, without having to resort to re-programming collectives using non-blocking point-to-point operations, as evidenced in High-performance Linpack benchmark (HPL) [9].…”

Section: A Mpi Collective Operations: Past Present and Futurementioning

confidence: 99%

“…Thus, enabling overlap of computation and communication. In addition, non-blocking collectives can also help mitigate OS-noise issues, that can often impact performance at scale [3], [8].…”

Section: Introductionmentioning

confidence: 99%

Design and Evaluation of Generalized Collective Communication Primitives with Overlap Using ConnectX-2 Offload Engine

Subramoni

Kandalla

Sur

et al. 2010

2010 18th IEEE Symposium on High Performance Interconnects

View full text Add to dashboard Cite

Collective communication operations provided by TheMessage Passing Interface (MPI) are heavily used by scientific applications at large scale. The current MPI standard, MPI-2.2, only defines blocking collective communication calls and it is expected that MPI-3 will allow for non-blocking collective communication. While it is possible to allow simultaneous computation and communication through thread-based designs, resource sharing across the threads is always a concern. The newly introduced ConnectX-2 InfiniBand adapter from Mellanox features an offload mechanism that enables the Network Interface Card (NIC) to perform a series of communication and reduction operations without the involvement of the host processor. Current generation MPI stacks implement each collective operation using point-to-point operations.To take advantage of offload feature in a rapidly changing architectural environment for all MPI collectives, they must be re-designed using flexible and generalized primitives. The primitives can then be used to compose various collective algorithms. The primitives must provide increased overlap with adapters supporting offload capabilities with varying collective group sizes and communication message sizes. In this paper, we take on the challenge of designing collective communication primitives with good overlap characteristics and evaluate their performance using ConnectX-2 offload feature. We also show how collectives such as Barrier can be designed using our communication primitives. Our evaluation reveals that we can achieve near perfect (94% -100%) overlap of computation and communication by using our primitives. Additionally, we Obsidian, QLogic, and Sun Microsystems. observe performance improvement of up to 5% using the Recv-Replicate primitive for data transfer.

show abstract

The impact of network noise at large-scale communication performance

Abstract: Abstract

Cited by 28 publications

References 24 publications

Performance Modeling and Comparative Analysis of the MILC Lattice QCD Application su3_rmd

Performance Modeling and Comparative Analysis of the MILC Lattice QCD Application su3_rmd

Award ER25844: Minimizing System Noise Effects for Extreme-Scale Scientific Simulation Through Function Delegation

Design and Evaluation of Generalized Collective Communication Primitives with Overlap Using ConnectX-2 Offload Engine

Contact Info

Product

Resources

About