Dynamic steering of HPC scientific workflows: A survey

Mattoso, Marta; Dias, Jonas; Ocaña, Kary; Ogasawara, Eduardo; Costa, Fernando José Pereira da; Horta, Felipe; Silva, Vítor; Oliveira, Daniel de

doi:10.1016/j.future.2014.11.017

Cited by 61 publications

(48 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This way, it is fundamental for scientists of any domain to be aware of the execution status in order to analyze if the current execution complies with some pre-defined quality and performance criteria. Based on this information steering [12], scientists may decide if they have to interfere in the execution (also known as dynamic workflows) or pause the workflow execution to change parameter values or even change the original workflow specification. These facilities allow for scientists to perform debugging, partial results analysis or to identify failures as early as they happen as online analysis of scientific workflows.…”

Section: Some Results: Uncertainty Quantification In Seismic Imaging mentioning

confidence: 99%

See 1 more Smart Citation

Enhancing Energy Production with Exascale HPC Methods

Mayo-García

Camata

María

et al. 2017

Communications in Computer and Information Science

Self Cite

View full text Add to dashboard Cite

Abstract. High Performance Computing (HPC) resources have become the key actor for achieving more ambitious challenges in many disciplines. In this step beyond, an explosion on the available parallelism and the use of special purpose processors are crucial. With such a goal, the HPC4E project applies new exascale HPC techniques to energy industry simulations, customizing them if necessary, and going beyond the state-of-the-art in the required HPC exascale simulations for different energy sources. In this paper, a general overview of these methods is presented as well as some specific preliminary results.

show abstract

Section: Some Results: Uncertainty Quantification In Seismic Imaging mentioning

confidence: 99%

“…With this purpose, Chiron uses workflow algebra SciWfA [12] to rule all workflow activities and represents consumed and produced data as relations. This workflow algebra uses a set of operators (Map, Filter, Reduce, SplitMap, SRQuery, and MRQuery) to be associated with workflow activities.…”

Section: Chiron Swfms For Managing Provenance In Uq Domainmentioning

confidence: 99%

Enhancing Energy Production with Exascale HPC Methods

Mayo-García

Camata

María

et al. 2017

Communications in Computer and Information Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…Netto et al [16] introduced a scheduler system to automatically offer more resources to parametric application jobs based on the quality of their intermediate generated results so as users could get faster to their desired goal. More recently, Mattoso et al [17] surveyed the use of steering in the context of HPC scientific workflows highlighting a tighter integration between the user and the underlying workflow execution system. Workflow management systems [18], [19], [20], [21] are also relevant to help users in complex scientific experiments.…”

Section: Related Workmentioning

confidence: 99%

JobPruner: A machine learning assistant for exploring parameter spaces in HPC applications

Silva

Netto

Cunha

2018

Future Generation Computer Systems

View full text Add to dashboard Cite

Abstract-High Performance Computing (HPC) applications are essential for scientists and engineers to create and understand models and their properties. These professionals depend on the execution of large sets of computational jobs that explore combinations of parameter values. Avoiding the execution of unnecessary jobs brings not only speed to these experiments, but also reductions in infrastructure usageparticularly important due to the shift of these applications to HPC cloud platforms. Our hypothesis is that data generated by these experiments can help users in identifying such jobs. To address this hypothesis we need to understand the similarity levels among multiple experiments necessary for job elimination decisions and the steps required to automate this process. In this paper we present a study and a machine learning-based tool called JobPruner to support parameter exploration in HPC experiments. The tool was evaluated with three real-world use cases from different domains including seismic analysis and agronomy. We observed the tool reduced 93% of jobs in a single experiment, while improving quality in most scenarios. In addition, reduction in job executions was possible even considering past experiments with low correlations.

show abstract

“…Provenance data, which is used for SWf analysis and SWf reproducibility, may be as important as the scientifc experiment itself [23]. The provenance data is typically stored in a database to provide on-line provenance query [28], and contains the information regarding activities, tasks and files. During the execution of a task, there may be multiple exchanges of provenance data between the computing node and the provenance database.…”

Section: Introductionmentioning

confidence: 99%

Scientific Workflow Scheduling with Provenance Data in a Multisite Cloud

Liu

Pacitti

Valduriez

et al. 2017

Transactions on Large-Scale Data- And Knowledge-Centered Systems XXXIII

Self Cite

View full text Add to dashboard Cite

Abstract. Recently, some Scientific Workflow Management Systems (SWfMSs) with provenance support (e.g. Chiron) have been deployed in the cloud. However, they typically use a single cloud site. In this paper, we consider a multisite cloud, where the data and computing resources are distributed at different sites (possibly in different regions). Based on a multisite architecture of SWfMS, i.e. multisite Chiron, and its provenance model, we propose a multisite task scheduling algorithm that considers the time to generate provenance data. We performed an extensive experimental evaluation of our algorithm using Microsoft Azure multisite cloud and two real-life scientific workflows (Buzz and Montage). The results show that our scheduling algorithm is up to 49.6% better than baseline algorithms in terms of total execution time.

show abstract

Dynamic steering of HPC scientific workflows: A survey

Cited by 61 publications

References 62 publications

Enhancing Energy Production with Exascale HPC Methods

Enhancing Energy Production with Exascale HPC Methods

JobPruner: A machine learning assistant for exploring parameter spaces in HPC applications

Scientific Workflow Scheduling with Provenance Data in a Multisite Cloud

Contact Info

Product

Resources

About