Towards long term data quality in a large scale biometrics experiment

Bui, Hoang; Wright, Diane; Helm, Clarence; Witty, Rachel; Flynn, Patrick J.; Thain, Douglas

doi:10.1145/1851476.1851559

Cited by 2 publications

(2 citation statements)

References 11 publications

(8 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All‐Pairs will manage an efficient spanning tree distribution of the dataset to support parallel comparisons. This approach has been used in biometrics research to perform comparison functions across every pair of subjects.…”

Section: Related Workmentioning

confidence: 99%

Balancing push and pull in Confuga, an active storage cluster file system for scientific workflows

Donnelly

Thain

2016

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

SUMMARYMost big-data analysis systems require users to adopt restricted abstractions to achieve scaling and system stability. While highly effective at establishing data locality and eliminating interdependencies, this approach is not easily incorporated into scientific workflows that are often complex and irregular graphs of sequential programs with multiple dependencies. To address this, we have developed an active storage cluster file system named Confuga which harnesses the file information already available in the workflow to enable efficient and controlled distribution of dependencies across active storage nodes. Confuga is built upon the idea of leveraging a job's namespace to eliminate unknown transfers and to plan the replication of all job dependencies. Replication is carried out through two opposing transfer methodologies: centrally managed push transfers and distributed pulls. We evaluate the effectiveness of the two transfer mechanisms using workflows that stress the ability of the cluster to replicate dependencies. Ultimately, we show that a balance of the two approaches achieves optimal file distribution. This is shown in two bioinformatics workflows where a careful balance of the two mechanisms leads to 48% and 77% improvements over only push or pull.

show abstract

Section: Related Workmentioning

confidence: 99%

Balancing push and pull in Confuga, an active storage cluster file system for scientific workflows

Donnelly

Thain

2016

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“…One of the key user interactions with the BXGrid Web portal is viewing snapshots of the collected image and video data. This allows researchers to quickly and easily enroll, verify, and modify data in the repository for purposes of maintaining data quality and integrity .…”

Section: Applicationsmentioning

confidence: 99%

Scripting distributed scientific workflows using Weaver

Bui

Thrasher

et al. 2011

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

SUMMARY Weaver is a high‐level distributed computing framework that enables researchers to construct scalable scientific data‐processing workflows. Instead of developing a new workflow language, we introduce a domain‐specific language built on top of Python called Weaver, which takes advantage of users' familiarity with the programming language, minimizes barriers to adoption, and allows for integration with a rich ecosystem of existing software. In this paper, we provide an overview of Weaver's programming model, which allows users to organize and specify scientific workflows by using a collection of datasets, functions, and abstractions. We also explain how these workflow specifications are compiled into a directed acyclic graph that is used by the Makeflow workflow manager to dispatch work to a variety of distributed execution platforms. To demonstrate the power and benefits of using the framework in constructing scientific research applications, the paper examines four distinct real‐world applications scripted using Weaver and analyzes the performance, scalability, and impact of the distributed generated scientific workflows. Copyright © 2011 John Wiley & Sons, Ltd.

show abstract

Towards long term data quality in a large scale biometrics experiment

Cited by 2 publications

References 11 publications

Balancing push and pull in Confuga, an active storage cluster file system for scientific workflows

Balancing push and pull in Confuga, an active storage cluster file system for scientific workflows

Scripting distributed scientific workflows using Weaver

Contact Info

Product

Resources

About