Understanding and Analyzing Interconnect Errors and Network Congestion on a Large Scale HPC System

Kumar, Mohit; Gupta, Saurabh; Patel, Tirthak; Wilder, Michael; Shi, Weisong; Fu, Song; Engelmann, Christian; Tiwari, Devesh

doi:10.1109/dsn.2018.00023

Cited by 10 publications

(1 citation statement)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [39], the authors proposed a novel scheme that used the spatial locality of failures to provide better application and system performance. In [40], the authors developed a thorough understanding of interconnect errors and job characteristics on an enterprise class supercomputer. Network traffic and access patterns were identified by Google's Network Telemetry [41].…”

Section: Related Workmentioning

confidence: 99%

Failure Diagnosis for Cluster Systems using Partial Correlations

Chuah¹,

Jhumka²,

Alt³

et al. 2021

2021 IEEE Intl Conf on Parallel &Amp; Distributed Processing With Applications, Big Data &Amp; Cloud Computing, Sustainable Com

View full text Add to dashboard Cite

Failures have expensive implications in HPC (High-Performance Computing) systems. Consequently, effective diagnosis of system failures is desired to help improve system reliability from both a remedial and preventive perspective. As HPC systems conduct extensive logging of resource usage and system events, parsing this data is an oft advocated basis for failure diagnosis. However, the high levels of concurrency that exist in HPC systems cause system events to frequently interleave in time and, as such, certain interactions appear or become indirect. which will be missed by current failure diagnostics techniques. To help uncover such indirect interactions, in this paper, we develop a novel approach that leverages the concept of partial correlation. The novel failure diagnostics workflow -called IFADE -extracts partial correlation of resource use counters and partial correlation of system errors. As part of our contributions, we (a) compare our diagnostics approach with current ones, (b) identify two previously unknown causes of system failures, validated by system designers and (c) provide insights into Lustre I/O and segmentation faults. IFADE has been put on the public domain to support system administrators in failure diagnosis.

show abstract

Section: Related Workmentioning

confidence: 99%