Workload Characteristics of a Multi-cluster Supercomputer

Li, Hui; Groep, David; Wolters, Lex

doi:10.1007/11407522_10

Cited by 86 publications

(100 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Different from the conclusions in previous research that a single distribution, such as the log-uniform distribution discussed in [10], can be used to present the queuing time distribution in a small system, for the workload of Kraken, a petascale supercomputer, any single distribution does not well fit to the job queuing time. In this study, the Gaussian mixture model (GMM), with two Gaussian component distributions, is applied to model the logarithm of the job queuing time, as illustrated in Figure 8-(1, 1).…”

Section: Individual Attribute Characteristicsmentioning

confidence: 70%

“…Similarly, the distribution of the actual job runtime, denoted as T r , cannot be simply modeled with a single distribution as well, such as the log-uniform [10], hypergamma [12], or Webull [1] distributions. In this study, we apply the GMM model to present the distribution of the logarithm of the actual job runtime, as shown in Figure 8- (2, 2).…”

Section: Individual Attribute Characteristicsmentioning

confidence: 99%

“…Using historical data of workload traces that were recorded on real machines, statistical analysis of workloads was performed to understand the characteristics, such as distributions of job runtime and memory usage, of a single HPC system [3], a multi-cluster supercomputer [8], or a Grid computing environment [2]. Statistical workload models were also widely studied to generate abstract representation of job attributes, such as fitting distributions to job attributes [17] and modeling correlations between job attribute pairs [7].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Comprehensive Workload Analysis and Modeling of a Petascale Supercomputer

You

Zhang

2013

Job Scheduling Strategies for Parallel Processing

View full text Add to dashboard Cite

Abstract. The performance of supercomputer schedulers is greatly affected by the characteristics of the workload it serves. A good understanding of workload characteristics is always important to develop and evaluate different scheduling strategies for an HPC system. In this paper, we present a comprehensive analysis of the workload characteristics of Kraken, the world's fastest academic supercomputer and 11th on the latest Top500 list, with 112,896 compute cores and peak performance of 1.17 petaflops. In this study, we use twelve-month workload traces gathered on the system, which include around 700 thousand jobs submitted by more than one thousand users from 25 research areas. We investigate three categories of the workload characteristics: 1) general characteristics, including distribution of jobs over research fields and different queues, distribution of job size for an individual user, job cancellation rate, job termination rate, and walltime request accuracy; 2) temporal characteristics, including monthly machine utilization, job temporal distributions for different time periods, job inter-arrival time between temporally adjacent jobs and jobs submitted by the same user; 3) execution characteristics, including distributions of each job attribute, such as job queuing time, job actual runtime, job size, and memory usage, and the correlations between these job attributes. This work provides a realistic basis for scheduler design and comparison by studying the supercomputer's workload with new approaches such as using Gaussian mixture model, and new viewpoints such as from the perspective of user community. To the best of our knowledge, it's the first research to systematically investigate the workload characteristics of a petascale supercomputer that is dedicated to open scientific research.

show abstract

Section: Individual Attribute Characteristicsmentioning

confidence: 70%

Section: Individual Attribute Characteristicsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Comprehensive Workload Analysis and Modeling of a Petascale Supercomputer

You

Zhang

2013

Job Scheduling Strategies for Parallel Processing

View full text Add to dashboard Cite

show abstract

“…Most of them are for single supercomputers not multi-cluster systems which we consider in this study. Logs from the Distributed ASCI Supercomputer [26] were archived in [29] and analyzed in [17]. Co-allocation details however were not included in the study and the jobs on the different clusters of DAS were archived separately.…”

Section: Job Streammentioning

confidence: 99%

Group-Wise Performance Evaluation of Processor Co-allocation in Multi-cluster Systems

Ngubiri

Vliet

2008

Job Scheduling Strategies for Parallel Processing

View full text Add to dashboard Cite

Abstract. Performance evaluation in multi-cluster processor co-allocation -like in many other parallel job scheduling problems-is mostly done by computing the average metric value for the entire job stream. This does not give a comprehensive understanding of the relative performance of the different jobs grouped by their characteristics. It is however the characteristics that affect how easy/hard jobs are to schedule. We, therefore, do not get to understand scheduler performance at job type level. In this paper, we study the performance of multi-cluster processor co-allocation for different job groups grouped by their size, components and widest component. We study their relative performance, sensitivity to parameters and how their performance is affected by the heuristics used to break them up into components. We show that the widest component us characteristic that most affects job schedulability. We also show that to get better performance, jobs should be broken up in such a way that the width of the widest component is minimized.

show abstract

“…To overcome this difficulty, many researchers decide to use randomly generated workloads in their work. These workloads are usually unrealistic because several statistical studies [7], [8], [13], [14], [15], [31] have shown that the characteristics of parallel system workloads 1 are far from independently and identically distributed. Instead, they have several important and correlated characteristics such as long range dependence, burstiness, and bag-of-tasks (BoT) behaviour.…”

Section: Introductionmentioning

confidence: 99%

A Realistic Integrated Model of Parallel System Workloads

Minh

Wolters

Epema

2010

2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing

Self Cite

View full text Add to dashboard Cite

Abstract-Performance evaluation is a significant step in the study of scheduling algorithms in large-scale parallel systems ranging from supercomputers to clusters and grids. One of the key factors that have a strong effect on the evaluation results is the workloads (or traces) used in experiments. In practice, several researchers use unrealistic synthetic workloads in their scheduling evaluations because they lack models that can help generate realistic synthetic workloads. In this paper we propose a full model to capture the following characteristics of real parallel system workloads: 1) long range dependence in the job arrival process, 2) temporal and spatial burstiness, 3) bag-oftasks behaviour, and 4) correlation between the runtime and the number of processors. Validation of our model with real traces shows that our model not only captures the above characteristics but also fits the marginal distributions well. In addition, we also present an approach to quantify burstiness in a job arrival process (temporal) as well as burstiness in the load of a trace (spatial).

show abstract

Workload Characteristics of a Multi-cluster Supercomputer

Cited by 86 publications

References 15 publications

Comprehensive Workload Analysis and Modeling of a Petascale Supercomputer

Comprehensive Workload Analysis and Modeling of a Petascale Supercomputer

Group-Wise Performance Evaluation of Processor Co-allocation in Multi-cluster Systems

A Realistic Integrated Model of Parallel System Workloads

Contact Info

Product

Resources

About