WorkWays: Interacting with Scientific Workflows

Nguyen, Hoang Anh; Abramson, David; Kiporous, Timoleon; Janke, Andrew L.; Galloway, Graham J.

doi:10.1109/gce.2014.6

Cited by 11 publications

(10 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In this work, we mostly focused on the first three, considering human adaptation as the core of computational steering and the one we mostly contributed with. As most related contributions to putting the human in the loop of HPC workflows [2,26,27], we focused on the efforts for engineering the backend enabling technology for user steering in an HPC workflow. More specifically, we contributed with allowing users to steer data reductions in scientific workflows online, focusing on providing a consistent execution within a data reduction, managing provenance data of user steering actions, and minimizing performance overheads in the HPC system (discussed in Section 7.4).…”

Section: Using D-chironmentioning

confidence: 99%

Data reduction in scientific workflows using provenance monitoring and user steering

Souza

Silva

Coutinho

et al. 2020

Future Generation Computer Systems

View full text Add to dashboard Cite

Scientific workflows need to be iteratively, and often interactively, executed for large input datasets. Reducing data from input datasets is a powerful way to reduce overall execution time in such workflows. When this is accomplished online (i.e., without requiring the user to stop execution to reduce the data, and then resume), it can save much time. However, determining which subsets of the input data should be removed becomes a major problem. A related problem is to guarantee that the workflow system will maintain execution and data consistent with the reduction. Keeping track of how users interact with the workflow is essential for data provenance purposes. In this paper, we adopt the "human-in-the-loop" approach, which enables users to steer the running workflow and reduce subsets from datasets online. We propose an adaptive workflow monitoring approach that combines provenance data monitoring and computational steering to support users in analyzing the evolution of key parameters and determining the subset of data to remove. We extend a provenance data model to keep track of users' interactions when they reduce data at runtime. In our experimental validation, we develop a test case from the oil and gas domain, using a 936-cores cluster. The results on this test case show that the approach yields reductions of 32% of execution time and 14% of the data processed.

show abstract

Section: Using D-chironmentioning

confidence: 99%

Data reduction in scientific workflows using provenance monitoring and user steering

Souza

Silva

Coutinho

et al. 2020

Future Generation Computer Systems

View full text Add to dashboard Cite

show abstract

“…In [49] a modelas-you-go approach has been proposed to enhance the classical workflow approach when tackling scientifical simulation challenges and the need for adaptation during workflow execution. WorkWays [40] allows for human-in-the-loop interaction and has already implemented a project involving CFD. SWAMP [54] is a Scientific Workflow System built entirely on composition of web services.…”

Section: Workflows Systems Science Gateways and Lincosimmentioning

confidence: 99%

LincoSim: a Web Based HPC-Cloud Platform for Automatic Virtual Towing Tank Analysis

Salvadore

Ponzini

2019

J Grid Computing

View full text Add to dashboard Cite

Thanks to evolving web technologies, computational platforms, automation tools and opensource software business model, today, it is possible to develop powerful automatic and virtualized web services for complex physical problems in engineering and design. In particular, in this work, we are presenting a new web based HPC-cloud platform for automatic virtual towing tank analysis. It is well known that the design project of a new hull requires a continuous integration of shape hypothesis and hydrodynamics verifications using analytical tools, 3D computational methods, experimental facilities and sea keeping trial tests. The complexity and the cost of such design tools increase considerably moving from analytical tools to sea keeping trials. In order to perform a meaningful trade-off between costs and high quality data acquiring, during the last decade the usage of 3D computational models has grown pushed also by well-known technological factors. Nevertheless, in the past, there were several limiting factors on the wide diffusion of 3D computational models to perform

show abstract

“…Researchers have also explored visualization techniques to add humans to the optimization process [28], [29], [30], [31]. WorkWays [32], [33] is a science gateway with human-in-the-loop support for running and managing scientific workflows.…”

Section: Related Workmentioning

confidence: 99%

JobPruner: A machine learning assistant for exploring parameter spaces in HPC applications

Silva

Netto

Cunha

2018

Future Generation Computer Systems

View full text Add to dashboard Cite

Abstract-High Performance Computing (HPC) applications are essential for scientists and engineers to create and understand models and their properties. These professionals depend on the execution of large sets of computational jobs that explore combinations of parameter values. Avoiding the execution of unnecessary jobs brings not only speed to these experiments, but also reductions in infrastructure usageparticularly important due to the shift of these applications to HPC cloud platforms. Our hypothesis is that data generated by these experiments can help users in identifying such jobs. To address this hypothesis we need to understand the similarity levels among multiple experiments necessary for job elimination decisions and the steps required to automate this process. In this paper we present a study and a machine learning-based tool called JobPruner to support parameter exploration in HPC experiments. The tool was evaluated with three real-world use cases from different domains including seismic analysis and agronomy. We observed the tool reduced 93% of jobs in a single experiment, while improving quality in most scenarios. In addition, reduction in job executions was possible even considering past experiments with low correlations.

show abstract

WorkWays: Interacting with Scientific Workflows

Cited by 11 publications

References 50 publications

Data reduction in scientific workflows using provenance monitoring and user steering

Data reduction in scientific workflows using provenance monitoring and user steering

LincoSim: a Web Based HPC-Cloud Platform for Automatic Virtual Towing Tank Analysis

JobPruner: A machine learning assistant for exploring parameter spaces in HPC applications

Contact Info

Product

Resources

About