Application monitoring and checkpointing in HPC

Jones, William M.; Daly, John T.; DeBardeleben, Nathan

doi:10.1145/2184512.2184574

Cited by 20 publications

(12 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Exascale requires new monitoring techniques, such as sub-optimal period scheduling [13] and strong usage of HPC system statistics [14] to improve system utilization. Thus, the effort on monitors development has been continuous in HPC systems.…”

Section: Related Workmentioning

confidence: 99%

Performance-Aware Scheduling of Parallel Applications on Non-Dedicated Clusters

2019

View full text Add to dashboard Cite

This work presents a HPC framework that provides new strategies for resource management and job scheduling, based on executing different applications in shared compute nodes, maximizing platform utilization. The framework includes a scalable monitoring tool that is able to analyze the platform’s compute node utilization. We also introduce an extension of CLARISSE, a middleware for data-staging coordination and control on large-scale HPC platforms that uses the information provided by the monitor in combination with application-level analysis to detect performance degradation in the running applications. This degradation, caused by the fact that the applications share the compute nodes and may compete for their resources, is avoided by means of dynamic application migration. A description of the architecture, as well as a practical evaluation of the proposal, shows significant performance improvements up to 20% in the makespan and 10% in energy consumption compared to a non-optimized execution.

show abstract

Section: Related Workmentioning

confidence: 99%

Performance-Aware Scheduling of Parallel Applications on Non-Dedicated Clusters

2019

View full text Add to dashboard Cite

show abstract

“…Under such circumstances, a decoupled storage system (e.g. a parallel file system such as GPFS [3]) does not provide sufficient I/O bandwidth to handle the explosion of data sizes: for example, Jones et al [4] predict dump times in the order of several hours.…”

Section: Introductionmentioning

confidence: 99%

Leveraging Naturally Distributed Data Redundancy to Reduce Collective I/O Replication Overhead

Nicolae

2015

2015 IEEE International Parallel and Distributed Processing Symposium

View full text Add to dashboard Cite

Dumping large amounts of related data simultaneously to local storage devices instead of a parallel file system is a frequent I/O pattern of HPC applications running at large scale. Since local storage resources are prone to failures and have limited potential to serve multiple requests in parallel, techniques such as replication are often used to enable resilience and high availability. However, replication introduces overhead, both in terms of network traffic necessary to distribute replicas, as well as extra storage space requirements. To reduce this overhead, state-of-art techniques often apply redundancy elimination (e.g. compression or deduplication) before replication, ignoring the natural redundancy that is already present. By contrast, this paper proposes a novel scheme that treats redundancy elimination and replication as a single co-optimized phase: remotely duplicated data is detected and directly leveraged to maintain a desired replication factor by keeping only as many replicas as needed and adding more if necessary. In this context, we introduce a series of high performance algorithms specifically designed to operate under tight and controllable constrains at large scale. We present how this idea can be leveraged in practice and demonstrate its viability for two real-life HPC applications.

show abstract

“…Under such circumstances, a decoupled storage system (e.g. a parallel file system such as GPFS [3] or a specialized storage system such as BlobSeer [4]) does not provide sufficient I/O bandwidth to handle the explosion of data sizes: for example, Jones et al [5] predict dump times in the order of several hours.…”

Section: Introductionmentioning

confidence: 99%

Techniques to improve the scalability of collective checkpointing at large scale

Nicolae

2015

2015 International Conference on High Performance Computing &Amp; Simulation (HPCS)

View full text Add to dashboard Cite

Application monitoring and checkpointing in HPC

Cited by 20 publications

References 13 publications

Performance-Aware Scheduling of Parallel Applications on Non-Dedicated Clusters

Performance-Aware Scheduling of Parallel Applications on Non-Dedicated Clusters

Leveraging Naturally Distributed Data Redundancy to Reduce Collective I/O Replication Overhead

Techniques to improve the scalability of collective checkpointing at large scale

Contact Info

Product

Resources

About