Towards supporting the life cycle of large scale scientific experiments

Mattoso, Marta; Werner, Cláudia Maria Lima; Travassos, Guilherme Horta; Braganholo, Vanessa; Ogasawara, Eduardo; Oliveira, Daniel de; Cruz, Sérgio Manuel Serra da; Martinho, Wallace; Murta, Leonardo

doi:10.1504/ijbpim.2010.033176

Cited by 94 publications

(69 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…A large set of sample workflows is needed for one to gain experience in configuring the workflow tasks flow [18]. These repositories, such as myExperiment [26,27] and CrowdLabs [28], contain hundreds of workflows with different goals.…”

Section: Related Workmentioning

confidence: 99%

“…Some of them focus on workflow reuse by providing composition at a high abstraction level [12,18,20,22], automatically generating the executable workflow. Others generate the workflow from execution trace logs and retrospective provenance [33][34][35].…”

Section: Our Previous Approachmentioning

confidence: 99%

“…However, they do not deal with the problem of suggesting new tasks to the user during workflow design. Other existing works [4,[11][12][13][14][15][16][17][18][19][20][21][22][23] propose the use of recommendation systems for scientific workflow design. However, most approaches miss the very sequential nature of workflows.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Improving workflow design by mining reusable tasks

Tosta¹,

Braganholo

Murta

et al. 2015

J Braz Comput Soc

Self Cite

View full text Add to dashboard Cite

Background: With the increasing popularity of scientific workflow management systems (SWfMS), more and more workflow specifications are becoming available. Such specifications contain precious knowledge that can be reused to produce new workflows. It is a fact that provenance data can help reusing third party code. However, finding the dependencies among programs without the support of a tool is not a trivial activity and, in many cases, becomes a barrier to build more sophisticated models and analysis. Due to the huge number of task versions available and their configuration parameters, this activity is highly error prone and counterproductive. Methods: In this work, we propose workflow recommender (WR), a recommendation service that aims at suggesting frequent combinations of workflow tasks for reuse. It works similarly to an e-commerce application that applies data mining techniques to help users find items they would like to purchase, predicting a list based on other user's choices. Results: Our experiments show that our approach is effective both in terms of performance and precision of the results. Conclusions:The approach is general in the sense that it can be coupled to any SWfMS.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Our Previous Approachmentioning

confidence: 99%

See 1 more Smart Citation

Improving workflow design by mining reusable tasks

Tosta¹,

Braganholo

Murta

et al. 2015

J Braz Comput Soc

Self Cite

View full text Add to dashboard Cite

show abstract

“…Over the last years most of scientific research is becoming more and more dependent of high performance computing (HPC) environments such as clusters and computational grids [1,2]. Most of scientific research usually deals with large volume of data (structure and unstructured) that requires huge CPU power to produce a result in a reasonable time.…”

Section: Introduction *mentioning

confidence: 99%

Is Cloud Computing the Solution for Brazilian Researchers?

Oliveira¹,

Ogasawara²

2010

IJCA

Self Cite

View full text Add to dashboard Cite

Over the last decade Brazilian government has considerably increased the investment in Research and Development (R&D) and especially in computer science research. Although investments have increased, they are not enough to fulfill the computational needs for all Brazilian researchers since there are many universities and research centers that are not able to acquire a cluster. Cloud computing can play an important role in providing an adequate infrastructure for researchers to do their research and produce valuable results, empowering Brazil in the scientific arena. This paper presents a brief evaluation on how cloud computing can improve Brazilian research by providing an adequate computational infrastructure for researchers.

show abstract

“…Scientific workflows are typically defined, executed and monitored by Scientific Workflow Management Systems (SWfMS) [2,3]. Currently, several SWfMS are available, e.g.…”

mentioning

confidence: 99%

Evaluating parameter sweep workflows in high performance computing

Chirigati

Silva

Ogasawara

et al. 2012

Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies

Self Cite

View full text Add to dashboard Cite

Scientific experiments based on computer simulations can be defined, executed and monitored using Scientific Workflow Management Systems (SWfMS). Several SWfMS are available, each with a different goal and a different engine. Due to the exploratory analysis, scientists need to run parameter sweep (PS) workflows, which are workflows that are invoked repeatedly using different input data. These workflows generate a large amount of tasks that are submitted to High Performance Computing (HPC) environments. Different execution models for a workflow may have significant differences in performance in HPC. However, selecting the best execution model for a given workflow is difficult due to the existence of many characteristics of the workflow that may affect the parallel execution. We developed a study to show performance impacts of using different execution models in running PS workflows in HPC. Our study contributes by presenting a characterization of PS workflow patterns (the basis for many existing scientific workflows) and its behavior under different execution models in HPC. We evaluated four execution models to run workflows in parallel. Our study measures the performance behavior of small, large and complex workflows among the evaluated execution models. The results can be used as a guideline to select the best model for a given scientific workflow execution in HPC. Our evaluation may also serve as a basis for workflow designers to analyze the expected behavior of an HPC workflow engine based on the characteristics of PS workflows.

show abstract

Towards supporting the life cycle of large scale scientific experiments

Cited by 94 publications

References 40 publications

Improving workflow design by mining reusable tasks

Improving workflow design by mining reusable tasks

Is Cloud Computing the Solution for Brazilian Researchers?

Evaluating parameter sweep workflows in high performance computing

Contact Info

Product

Resources

About