Comprehensive Workload Analysis and Modeling of a Petascale Supercomputer

You, Haihang; Zhang, Hao

doi:10.1007/978-3-642-35867-8_14

Cited by 21 publications

(13 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Two-level scheduling models assume that the tasks run on them are largely short-lived and relinquish resource frequently, a valid assumption for data-intensive workloads [21], [22]. However, because HPC workloads are dominated by compute-bound and long-running applications [1], [23], two-level scheduling models are not applicable for HPC nodes.…”

Section: Related Workmentioning

confidence: 99%

“…Our simulators aim to address two main applications: data-intensive applications and traditional MPI applications. There are many approaches to partition workloads among clusters [1], [23]. To simplify and obtain the critical behavior characteristics of different scheduling models, we select a simple two-way split on top of our simulators between I/O-bound jobs that provide end-user operations (e.g., web services) and internal infrastructure services (e.g., BigTable), and compute-bound jobs that perform a computation.…”

Section: Workloadsmentioning

confidence: 99%

See 1 more Smart Citation

Improving Per-Node Computing Efficiency by an Adaptive Lock-Free Scheduling Model

Zheng

Zhou

et al. 2018

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

Job scheduling on many-core computers with tens or even hundreds of processing cores is one of the key technologies in High Performance Computing (HPC) systems. Despite many scheduling algorithms have been proposed, scheduling remains a challenge for executing highly effective jobs that are assigned in a single computing node with diverse scheduling objectives. On the other hand, the increasing scale and the need for rapid response to changing requirements are hard to meet with existing scheduling models in an HPC node. To address these issues, we propose a novel adaptive scheduling model that is applied to a single node with a many-core processor; this model solves the problems of scheduling efficiency and scalability through an adaptive optimistic control mechanism. This mechanism exposes information such that all the cores are provided with jobs and the tools necessary to take advantage of that information and thus compete for resources in an uncoordinated manner. At the same time, the mechanism is equipped with adaptive control, allowing it to adjust the number of running tools dynamically when frequent conflict happens. We justify this scheduling model and present the simulation results for synthetic and real-world HPC workloads, in which we compare our proposed model with two widely used scheduling models, i.e. multi-path monolithic and two-level scheduling. The proposed approach outperforms the other models in scheduling efficiency and scalability. Our results demonstrate that the adaptive optimistic control affords significant improvements for HPC workloads in the parallelism of the node-level scheduling model and performance.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Workloadsmentioning

confidence: 99%

Improving Per-Node Computing Efficiency by an Adaptive Lock-Free Scheduling Model

Zheng

Zhou

et al. 2018

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

show abstract

“…Since the dataintensive applications will converge with HPC applications in the future, our simulators aim at two main applications: data-intensive ones and traditional MPI ones [48]. There are many approaches to partitioning workloads among clusters [42,43,52,53]. To simplify and obtain the critical behavior characteristics of different schedulers, we select a simple twoway split on top of our simulators between I/O bound jobs that provide end-user operations (e.g., web services), internal infrastructure services (e.g., BigTable), and compute-bound jobs that perform a computation.…”

Section: Parameters For Simulation the Scheduler Decision Time (mentioning

confidence: 99%

“…As discussed [52,53], we assume that all tasks of jobs from HPC workloads are CPU-intensive and the tasks in a one-task job are sequential. For data-intensive jobs [43], we assume that a job is made up of one or more tasks (occasionally thousands of tasks), which are mostly CPU-intensive (> 80%) but with a fast turnaround, whereas the rest are I/O-intensive (< 20%) and consume the majority of resources.…”

Section: Workloads Workload Heterogeneitymentioning

confidence: 99%

Optimizing Job Coscheduling by Adaptive Deadlock-Free Scheduler

Zheng

et al. 2018

Mathematical Problems in Engineering

View full text Add to dashboard Cite

It is ubiquitous that multiple jobs coexist on the same machine, because tens or hundreds of cores are able to reside on the same chip. To run multiple jobs efficiently, the schedulers should provide flexible scheduling logic. Besides, corunning jobs may compete for the shared resources, which may lead to performance degradation. While many scheduling algorithms have been proposed for supporting different scheduling logic schemes and alleviating this contention, job coscheduling without performance degradation on the same machine remains a challenging problem. In this paper, we propose a novel adaptive deadlock-free scheduler, which provides flexible scheduling logic schemes and adopts optimistic lock control mechanism to coordinate resource competition among corunning jobs. This scheduler exposes all underlying resource information to corunning jobs and gives them necessary utensils to make use of that information to compete resource in a free-for-all manner. To further relieve performance degradation of coscheduling, this scheduler enables the automated control over the number of active utensils when frequent conflict becomes the performance bottleneck. We justify our adaptive deadlock-free scheduling and present simulation results for synthetic and real-world workloads, in which we compare our proposed scheduler with two prevalent schedulers. It indicates that our proposed approach outperforms the compared schedulers in scheduling efficiency and scalability. Our results also manifest that the adaptive deadlock-free control facilitates significant improvements on the parallelism of node-level scheduling and the performance for workloads.

show abstract

“…This study not only helps to understand the behavior of current HPC systems but can also be used to develop new scheduling strategies for the forthcoming exascale systems. Similar studies mixing the characterization, modeling, and prediction of HPC workloads have been proposed in [5,17]. In [8], the authors proposed a complete reconguration of their batch scheduling systems, including the denition of the scheduling queues, based on a thorough analysis of the workload characteristics and detailed simulations.…”

Section: Introductionmentioning

confidence: 96%

Improving Fairness in a Large Scale HTC System Through Workload Analysis and Simulation

Azevedo

Klusáček

Suter

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Monitoring and analyzing the execution of a workload is at the core of the operation of data centers. It allows operators to verify that the operational objectives are satised or detect and react to any unexpected and unwanted behavior. However, the scale and complexity of large workloads composed of millions of jobs executed each month on several thousands of cores, often limit the depth of such an analysis. This may lead to overlook some phenomena that, while not harmful at a global scale, can be detrimental to a specic class of users. In this paper, we illustrate such a situation by analyzing a large High Throughput Computing (HTC) workload trace coming from one of the largest academic computing centers in France. The Fair-Share algorithm at the core of the batch scheduler ensures that all user groups are fairly provided with an amount of computing resources commensurate to their expressed needs. However, a deeper analysis of the produced schedule, especially of the job waiting times, shows a certain degree of unfairness between user groups. We identify the conguration of the quotas and scheduling queues as the main root causes of this unfairness. We thus propose a drastic reconguration of the system that aims at being more suited to the characteristics of the workload and at better balancing the waiting time among user groups. We evaluate the impact of this reconguration through detailed simulations. The obtained results show that it still satises the main operational objectives while signicantly improving the quality of service experienced by formerly unfavored users.

show abstract

Comprehensive Workload Analysis and Modeling of a Petascale Supercomputer

Cited by 21 publications

References 16 publications

Improving Per-Node Computing Efficiency by an Adaptive Lock-Free Scheduling Model

Improving Per-Node Computing Efficiency by an Adaptive Lock-Free Scheduling Model

Optimizing Job Coscheduling by Adaptive Deadlock-Free Scheduler

Improving Fairness in a Large Scale HTC System Through Workload Analysis and Simulation

Contact Info

Product

Resources

About