Watch Out for the Bully! Job Interference Study on Dragonfly Network

Xu, Yang; Jenkins, John; Mubarak, Misbah; Ross, Robert; Lan, Zhiling

doi:10.1109/sc.2016.63

Cited by 57 publications

(29 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most solutions optimize job allocation to minimize the contention on the links, either on dragonfly networks [39] or on other topologies [11,38]. For example, Yang et al [45,47] show that whereas for communication intensive jobs a random allocation is more beneficial, for less communication-intensive jobs a contiguous allocation is better. Starting from this observation, they propose a hybrid allocation scheme, to allocate communication-intensive jobs randomly whereas the less communication-intensive jobs are allocated on contiguous nodes.…”

Section: Related Workmentioning

confidence: 99%

Mitigating network noise on Dragonfly networks through application-aware routing

Sensi

Girolamo

Hoefler

2019

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

System noise can negatively impact the performance of HPC systems, and the interconnection network is one of the main factors contributing to this problem. To mitigate this effect, adaptive routing sends packets on non-minimal paths if they are less congested. However, while this may mitigate interference caused by congestion, it also generates more traffic since packets traverse additional hops, causing in turn congestion on other applications and on the application itself. In this paper, we first describe how to estimate network noise. By following these guidelines, we show how noise can be reduced by using routing algorithms which select minimal paths with a higher probability. We exploit this knowledge to design an algorithm which changes the probability of selecting minimal paths according to the application characteristics. We validate our solution on microbenchmarks and real-world applications on two systems relying on a Dragonfly interconnection network, showing noise reduction and performance improvement.is feared by datacenter operators in so-called incast or hot-spot (many-to-one) patterns. In addition to this, adaptive routing may be affected by the so-called phantom congestion problem [46]. Namely, as congestion information is propagated with some delay, a node may react too late to congestion events. In this paper, we will be first to demonstrate and quantify the influence of different routing schemes on network noise in practice.Analyzing network noise in detail is delicate because in practice, when observing application delays, it is hard to distinguish between network noise, operating system noise, and application imbalance. Other works have used network counters but may run into the fallacy that correlation is not causation by ignoring the aspects of unrelated traffic. We will clarify several potential problems in investigations of network noise and develop a set of general guidelines for our analysis. Using these guidelines, we study the relationship between different adaptive routing schemes, application performance, and network noise. Our findings on two large-scale Cray Aries systems, Piz Daint and NERSC's Cori, are remarkable: not only will changing the adaptive routing mode reduce communication times and speed up applications up to twice, but it will also significantly reduce performance variation.We find that the best routing mode that minimizes network noise and maximizes performance depends not only on the characteristics of the allocation but also on the communication load. For example, large-scale alltoall communications are best routed with the default mode whereas many other communication patterns benefit from mostly minimal routing. Because this all depends on the location of the communication peers, there is no simple static rule to select the best routing mode. To address this, we develop a simple but effective dynamic routing library that observes the network state through local network counters and adjusts the routing mode for each message based on application characteristics s...

show abstract

Section: Related Workmentioning

confidence: 99%

Mitigating network noise on Dragonfly networks through application-aware routing

Sensi

Girolamo

Hoefler

2019

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

show abstract

“…Earlier studies showed that applications that run on Cray XC40 might experience a high degree of variability in execution time and performance [10,14,28]. (n, p).…”

Section: Measurements Variabilitymentioning

confidence: 99%

Understanding the Scalability of Molecular Simulation Using Empirical Performance Modeling

Shudler

Vrabec

Wolf

2019

Programming and Performance Visualization Tools

View full text Add to dashboard Cite

Molecular dynamics (MD) simulation allows for the study of static and dynamic properties of molecular ensembles at various molecular scales, from monatomics to macromolecules such as proteins and nucleic acids. It has applications in biology, materials science, biochemistry, and biophysics. Recent developments in simulation techniques spurred the emergence of the computational molecular engineering (CME) field, which focuses specifically on the needs of industrial users in engineering. Within CME, the simulation code ms2 allows users to calculate thermodynamic properties of bulk fluids. It is a parallel code that aims to scale the temporal range of the simulation while keeping the execution time minimal. In this paper, we use empirical performance modeling to study the impact of simulation parameters on the execution time. Our approach is a systematic workflow that can be used as a blueprint in other fields that aim to scale their simulation codes. We show that the generated models can help users better understand how to scale the simulation with minimal increase in execution time.

show abstract

“…However, while Aries manages long bouts of congestion better than Gemini does, application runtime variability due to network performance remains a concern [15]. • Detection of long-duration congestion using traffic measurements can facilitate intervention such as rank remapping or rescheduling of bully jobs [6]. The 99.9th percentile congested link duration observed in both systems for P T S th ≤ 20% is greater than a minute.…”

Section: • Impact Of Routing Algorithms On Congestion (See Subsectionmentioning

confidence: 99%

A Study of Network Congestion in Two Supercomputing High-Speed Interconnects

Jha

Patke

Brandt

et al. 2019

2019 IEEE Symposium on High-Performance Interconnects (HOTI)

View full text Add to dashboard Cite

Network congestion in high-speed interconnects is a major source of application runtime performance variation. Recent years have witnessed a surge of interest from both academia and industry in the development of novel approaches for congestion control at the network level and in application placement, mapping, and scheduling at the system-level. However, these studies are based on proxy applications and benchmarks that are not representative of field-congestion characteristics of high-speed interconnects. To address this gap, we present (a) an end-to-end framework for monitoring and analysis to support long-term field-congestion characterization studies, and (b) an empirical study of network congestion in petascale systems across two different interconnect technologies: (i) Cray Gemini, which uses a 3-D torus topology, and (ii) Cray Aries, which uses the DragonFly topology.

show abstract

Watch Out for the Bully! Job Interference Study on Dragonfly Network

Cited by 57 publications

References 22 publications

Mitigating network noise on Dragonfly networks through application-aware routing

Mitigating network noise on Dragonfly networks through application-aware routing

Understanding the Scalability of Molecular Simulation Using Empirical Performance Modeling

A Study of Network Congestion in Two Supercomputing High-Speed Interconnects

Contact Info

Product

Resources

About