Predicting Output Performance of a Petascale Supercomputer

Xie, Bing; Huang, Yezhou; Chase, Jeffrey S.; Choi, Jong Youl; Klasky, Scott; Lofstead, Jay; Oral, Sarp

doi:10.1145/3078597.3078614

Cited by 46 publications

(17 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Schmidt et al [28] proposed a prediction scheme using artificial neural network in HPC system. Other works [18,19,36] also tried to predict various performance metrics for large clusters. Similar to these studies, our scheme aims to predict the performance of the application using characteristics of the HPC environment.…”

Section: Prediction Using System Characteristicsmentioning

confidence: 99%

Towards HPC I/O Performance Prediction through Large-scale Log Analysis

Kim

Sim

et al. 2020

Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing

View full text Add to dashboard Cite

Large-scale high performance computing (HPC) systems typically consist of many thousands of CPUs and storage units, while used by hundreds to thousands of users at the same time. Applications from these large numbers of users have diverse characteristics, such as varying compute, communication, memory, and I/O intensiveness. A good understanding of the performance characteristics of each user application is important for job scheduling and resource provisioning. Among these performance characteristics, the I/O performance is difficult to predict because the I/O system software is complex, the I/O system is shared among all users, and the I/O operations also heavily rely on networking systems. To improve the prediction of the I/O performance on HPC systems, we propose to integrate information from a number of different system logs and develop a regression-based approach that dynamically selects the most relevant features from the most recent log entries, and automatically select the best regression algorithm for the prediction task. Evaluation results show that our proposed scheme can predict the I/O performance with up to 84% prediction accuracy in the case of the I/O-intensive applications using the logs from CORI supercomputer at NERSC.

show abstract

Section: Prediction Using System Characteristicsmentioning

confidence: 99%

Towards HPC I/O Performance Prediction through Large-scale Log Analysis

Kim

Sim

et al. 2020

Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing

View full text Add to dashboard Cite

show abstract

“…In [8], a performance prediction model is developed by developers that aims to improve job runtime estimation for better job scheduling. The authors use the property of static iterative scientific code to produce near constant I/O burst, when considered over a longer period of time.…”

Section: Related Workmentioning

confidence: 99%

Semi-automatic Assessment of I/O Behavior by Inspecting the Individual Client-Node Timelines—An Explorative Study on $$10^{6}$$ Jobs

Betke

Kunkel

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

HPC applications with suboptimal I/O behavior interfere with well-behaving applications and lead to increased application runtime. In some cases, this may even lead to unresponsive systems and unfinished jobs. HPC monitoring systems can aid users and support staff to identify problematic behavior and support optimization of problematic applications. The key issue is how to identify relevant applications? A profile of an application doesn't allow identifying problematic phases during the execution but tracing of each individual I/O is too invasive.In this work, we split the execution into segments, i.e., windows of fixed size and analyze profiles of them. We develop three I/O metrics to identify three relevant classes of inefficient I/O behaviors, and evaluate them on raw data of 1,000,000 jobs on the supercomputer Mistral. The advantages of our method is that temporal information about I/O activities during job runtime is preserved to some extent and can be used to identify phases of inefficient I/O.The main contribution of this work is the segmentation of time series and computation of metrics (Job-I/O-Utilization, Job-I/O-Problem-Time, and Job-I/O-Balance) that are effective to identify problematic I/O phases and jobs.

show abstract

“…Any non-straggler storage will become straggler within 10 minutes. Furthermore, in [45] authors show that while application is running, all I/O will trend to a long-term average performance for that sized I/O.…”

Section: Related Work and Comparisonmentioning

confidence: 99%

Client-side straggler-aware I/O scheduler for object-based parallel file systems

2019

View full text Add to dashboard Cite

Object-based parallel file systems have emerged as promising storage solutions for high-performance computing (HPC) systems. Despite the fact that object storage provides a flexible interface, scheduling highly concurrent I/O requests that access a large number of objects still remains as a challenging problem, especially in the case when stragglers (storage servers that are significantly slower than others) exist in the system. An efficient I/O scheduler needs to avoid possible stragglers to achieve low latency and high throughput. In this paper, we introduce a logassisted straggler-aware I/O scheduling to mitigate the impact of storage server stragglers. The contribution of this study is threefold. First, we introduce a client-side, log-assisted, straggler-aware I/O scheduler architecture to tackle the storage straggler issue in HPC systems. Second, we present three scheduling algorithms that can make efficient decision for scheduling I/Os while avoiding stragglers based on such an architecture. Third, we evaluate the proposed I/O scheduler using simulations, and the simulation results have confirmed the promise of the newly introduced straggler-aware I/O scheduler.

show abstract

Predicting Output Performance of a Petascale Supercomputer

Cited by 46 publications

References 33 publications

Towards HPC I/O Performance Prediction through Large-scale Log Analysis

Towards HPC I/O Performance Prediction through Large-scale Log Analysis

Semi-automatic Assessment of I/O Behavior by Inspecting the Individual Client-Node Timelines—An Explorative Study on $$10^{6}$$ Jobs

Client-side straggler-aware I/O scheduler for object-based parallel file systems

Contact Info

Product

Resources

About