Scripting distributed scientific workflows using Weaver

Bui, Peter; Yu, Li; Thrasher, Andrew; Carmichael, Rory; Lanc, Irena; Donnelly, Patrick J.; Thain, Douglas

doi:10.1002/cpe.1871

Cited by 5 publications

(4 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…easily switch between execution platforms and currently supports Condor [13,31], SGE [14], Work Queue [32], and other systems. To programmatically create large workflows and the workflows used in our experimental results, we use the Weaver [8] workflow compiler. Concurrent job execution in Confuga comes from dispatching jobs to multiple storage nodes.…”

Section: The Execution Modelmentioning

confidence: 99%

“…It is designed to easily switch between execution platforms and currently supports Condor , SGE , Work Queue , and other systems. To programmatically create large workflows and the workflows used in our experimental results, we use the Weaver workflow compiler.…”

Section: The Problem Settingmentioning

confidence: 99%

“…There have been efforts to make suitable data‐locality‐aware abstractions available to scientific workflows , but adoption is limited as abstractions impose certain structural constraints on the workflow. Instead, the scientific community has adopted a flexible workflow model composed of standard sequential applications chained together by data dependencies and represented as a directed acyclic graph (DAG) of jobs.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Balancing push and pull in Confuga, an active storage cluster file system for scientific workflows

Donnelly

Thain

2016

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

SUMMARYMost big-data analysis systems require users to adopt restricted abstractions to achieve scaling and system stability. While highly effective at establishing data locality and eliminating interdependencies, this approach is not easily incorporated into scientific workflows that are often complex and irregular graphs of sequential programs with multiple dependencies. To address this, we have developed an active storage cluster file system named Confuga which harnesses the file information already available in the workflow to enable efficient and controlled distribution of dependencies across active storage nodes. Confuga is built upon the idea of leveraging a job's namespace to eliminate unknown transfers and to plan the replication of all job dependencies. Replication is carried out through two opposing transfer methodologies: centrally managed push transfers and distributed pulls. We evaluate the effectiveness of the two transfer mechanisms using workflows that stress the ability of the cluster to replicate dependencies. Ultimately, we show that a balance of the two approaches achieves optimal file distribution. This is shown in two bioinformatics workflows where a careful balance of the two mechanisms leads to 48% and 77% improvements over only push or pull.

show abstract

Section: The Execution Modelmentioning

confidence: 99%

Section: The Problem Settingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Balancing push and pull in Confuga, an active storage cluster file system for scientific workflows

Donnelly

Thain

2016

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“…Because a single bioinformatics Makeflow can be thousands of lines long, we have shown that creating higher level tools are the key for successful and practical parallelization. To this end, we have shown the Weaver tool, together with the Starch archive manager, can provide a standardized framework for parallellized bioinformatics workflows .…”

Section: Introductionmentioning

confidence: 99%

Adapting bioinformatics applications for heterogeneous systems: a case study

Lanc

Bui

Thain

et al. 2012

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

The advent of new sequencing technologies has generated extremely large amounts of information. To successfully apply bioinformatics tools to such large datasets, they need to exhibit scalability and ideally elasticity in diverse computing environments. We describe the application of Weaver to the PEMer structural variation detection workflow. Because the original workflow has an intractable sequential running time on large datasets, it also has a batch implementation designed for a shared file system. Using scripts provided by the developers of PEMer, along with the Weaver Python module, the Starch archive generator, and the Makeflow workflow engine, we have refactored PEMer for elastic scaling on personal clouds. Our case study describes the various challenges faced when constructing such a workflow, from dealing with failure detection, to managing dependencies, to handling the quirks of the underlying operating systems. The practice of scaling bioinformatics tools is increasingly commonplace. As such, the hands-on application of refactoring techniques to PEMer can serve as a valuable guide for those looking to reconfigure other bioinformatics software. Significantly, our customized Makeflow framework enabled elastic deployment on a wider variety of systems while substantially reducing wall clock runtimes using hundreds of cores.

show abstract

Programming languages for data-Intensive HPC applications: A systematic mapping study

et al. 2020

View full text Add to dashboard Cite

We present the results of a systematic literature review that examines the main paradigms and properties of programming languages developed for and used in High Performance Computing for Big Data processing. The systematic literature review is based on a combination of automated keyword-based search in the Elsevier Science Direct database and further digital databases for articles published in international peer-reviewed journals and conferences, leading to an initial sample of 420 articles, which was then narrowed down in a second phase to 152 articles found relevant and published 2006-2018. The manual analysis of these articles allowed us to identify 26 languages used in 33 of these articles for HPC for Big Data processing. We analyzed the languages and their usage in these articles by 22 criteria and summarize the results in this article. We evaluate the outcomes of the literature review by comparing them with opinions of domain experts. Our results indicate that, for instance, the majority of the used HPC languages in the context of Big Data are text-based general-purpose programming languages and target the end-user community.

show abstract

Scripting distributed scientific workflows using Weaver

Cited by 5 publications

References 31 publications

Balancing push and pull in Confuga, an active storage cluster file system for scientific workflows

Balancing push and pull in Confuga, an active storage cluster file system for scientific workflows

Adapting bioinformatics applications for heterogeneous systems: a case study

Programming languages for data-Intensive HPC applications: A systematic mapping study

Contact Info

Product

Resources

About