A runtime estimation framework for ALICE

Pumma, Sarunya; Phunchongharn, Phond; Chapeland, S.; Feng, Wu-chun; Achalakul, Tiranee

doi:10.1016/j.future.2017.02.040

Cited by 14 publications

(22 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 1 summarizes major similarities and differences between our work and the existing studies. First, the examined papers concern various application domains: load sharing facility (LSF) [8], parallel program [9], [11], [25], cloud [10], [29], HPC [2], [14], [15], [19], location-based services [20]- [23], databases [26]- [28], big data applications [29], [30], and scientific workloads [12], [13], [16]- [19]. The runtime estimation problem addressed in this paper applies to the scientific workloads domain.…”

Section: Related Workmentioning

confidence: 99%

“…Some of these [26], [27], [29] developed (pure) analytical models and assess the validity of the model. Many of the existing studies [20]- [24], [28] build neural-net based learning models and utilize them for deriving estimated time; many others [2], [10], [16]- [19], [25], [30] use tree and linear regression based machine learning models. Some other works [11], [12], [14], [15] use hybrid methods combining these tools-analytical model, machine learning, and deep learning.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

CLUTCH: A Clustering-Driven Runtime Estimation Scheme for Scientific Simulations

Suh

Kim

2020

IEEE Access

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

CLUTCH: A Clustering-Driven Runtime Estimation Scheme for Scientific Simulations

Suh

Kim

2020

IEEE Access

View full text Add to dashboard Cite

“…Pumma et al [90] use an approach that completely ignores metadata. Instead, each job is run for a short while and profiled using platform independent low level features derived from hardware performance counters.…”

Section: Resource Usage Pattern Modeling At the Job Levelmentioning

confidence: 99%

Predictive performance modeling for distributed batch processing using black box monitoring and machine learning

Witt

Bux

Gusew

et al. 2019

Information Systems

View full text Add to dashboard Cite

In many domains, the previous decade was characterized by increasing data volumes and growing complexity of computational workloads, creating new demands for highly data-parallel computing in distributed systems. Effective operation of these systems is challenging when facing uncertainties about the performance of jobs and tasks under varying resource configurations, e. g., for scheduling and resource allocation. We survey predictive performance modeling (PPM) approaches to estimate performance metrics such as execution duration, required memory or wait times of future jobs and tasks based on past performance observations. We focus on non-intrusive methods, i. e., methods that can be applied to any workload without modification, since the workload is usually a black-box from the perspective of the systems managing the computational infrastructure. We classify and compare sources of performance variation, predicted performance metrics, required training data, use cases, and the underlying prediction techniques. We conclude by identifying several open problems and pressing research needs in the field.

show abstract

“…Many developments in performance prediction can be categorized into Regression [24], [25], [31], [32], [33], Classification [34], [35], Similarity based predictions [36], [37], and dynamic Time-Series predictions [26], [27], [38], [39], [40], [41] . We can also index these techniques on other dimensions, depending on priority of workflow application, such as ease of data collection for Machine Learning tech-niques, level of sophistication involved in instrumenting a resource for time-series based dynamic prediction, sourcecode evaluation for intrusive methods, and workload modeling methods for platforms with resource-contention.…”

Section: Related Workmentioning

confidence: 99%

“…[33] presents an on-line estimation scheme that uses correlations among data sets combined with clustering techniques for non-obvious scenarios, where correlations are hard to model. [35] deploys application-profiles to estimate run times, and tests on real data sets. This work samples workload, performs classification and leverages category specific model for run time prediction.…”

Section: Related Workmentioning

confidence: 99%

Toward a Methodology and Framework for Workflow-Driven Team Science

et al. 2019

View full text Add to dashboard Cite

Scientific workflows are powerful tools for management of scalable experiments, often composed of complex tasks running on distributed resources. Existing cyberinfrastructure provides components that can be utilized within repeatable workflows. However, data and computing advances continuously change the way scientific workflows get developed and executed, pushing the scientific activity to be more data-driven, heterogeneous and collaborative. Workflow development today depends on the effective collaboration and communication of a cross-disciplinary team, not only with humans but also with analytical systems and infrastructure. This paper presents a collaboration-centered reference architecture to extend workflow systems with dynamic, predictable and programmable interfaces to systems and infrastructure while bridging the exploratory and scalable activities in the scientific process. We also present a conceptual design towards the development of methodologies and tools for effective workflow-driven collaborations, namely the PPoDS methodology and the SmartFlows Toolkit for smart utilization of workflows in a rapidly evolving cyberinfrastructure ecosystem.

show abstract

A runtime estimation framework for ALICE

Cited by 14 publications

References 18 publications

CLUTCH: A Clustering-Driven Runtime Estimation Scheme for Scientific Simulations

CLUTCH: A Clustering-Driven Runtime Estimation Scheme for Scientific Simulations

Predictive performance modeling for distributed batch processing using black box monitoring and machine learning

Toward a Methodology and Framework for Workflow-Driven Team Science

Contact Info

Product

Resources

About