Scientific workflow design 2.0: Demonstrating streaming data collections in Kepler

Dou, Lei; Zinn, Daniel; McPhillips, Timothy M.; Köhler, Sven; Riddle, Sean; Bowers, Shawn; Ludäscher, Bertram

doi:10.1109/icde.2011.5767938

Cited by 17 publications

(5 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…How can workflow development become more like storytelling and less like programming? The COMAD model (Collection-Oriented Modeling and Design), implemented as a Kepler director [61,62,32], and the related VDAL model (Virtual Data Assembly Lines) [92,91] use a conveyorbelt assembly-line metaphor to create workflow designs that are mostly linear, and thus easier to understand and modify. Instead of using shims and complex wiring to ship just the required data fragments to only those actors where they are immediately needed, the conveyorbelt approach instead provides every actor with a subscription mechanism to "pick-up" only the relevant parts 3 A physical shim is a thin strip of metal for aligning pipes.…”

Section: Workflow Designmentioning

confidence: 99%

“…In this way, changes to those other parts, i.e., outside an actor's read scope, will not affect that actor's functionality, making the overall approach much more change resilient. As a result, even more so than on a physical assembly line, steps can be easily added, swapped, replaced, or removed in these approaches [62,91,32]: For example, Figure 2 shows a COMAD redesign of the workflow from Fig. 1.…”

Section: Workflow Designmentioning

confidence: 99%

See 1 more Smart Citation

Scientific Workflows and Provenance: Introduction and Research Opportunities

et al. 2012

Self Cite

View full text Add to dashboard Cite

Scientific workflows are becoming increasingly popular for compute-intensive and data-intensive scientific applications. The vision and promise of scientific workflows includes rapid, easy workflow design, reuse, scalable execution, and other advantages, e.g., to facilitate "reproducible science" through provenance (e.g., data lineage) support. However, as described in the paper, important research challenges remain. While the database community has studied (business) workflow technologies extensively in the past, most current work in scientific workflows seems to be done outside of the database community, e.g., by practitioners and researchers in the computational sciences and eScience. We provide a brief introduction to scientific workflows and provenance, and identify areas and problems that suggest new opportunities for database research.

show abstract

Section: Workflow Designmentioning

confidence: 99%

Section: Workflow Designmentioning

confidence: 99%

Scientific Workflows and Provenance: Introduction and Research Opportunities

et al. 2012

Self Cite

View full text Add to dashboard Cite

show abstract

“…Compared with workflows developed elsewhere, there are two human data curation steps. The Kurator workflow (Dou et al 2011(Dou et al , 2012 only contains one, and includes more automation. In crowdsourcing projects, the human curation steps have entirely been replaced by repeated transcription of the same samples (Flemons et al 2012;Hill et al 2012): When there are enough repetitions that match, the result is automatically accepted.…”

Section: Discussionmentioning

confidence: 99%

“…Dou et al 2011Dou et al , 2012Lehtonen et al 2011;Nelson et al 2012;Tegelberg et al 2012). The methods used in some of these steps may vary depending on specimen type.…”

Section: Introductionmentioning

confidence: 99%

DigiWeb - a workflow environment for quality assurance of transcription in digitization of natural history collections

et al. 2014

View full text Add to dashboard Cite

-Data produced by digitization increases the scientific use of natural history collections. However, in mass digitization, attention must be paid to the flawless management of the workflows, and high quantities of end results should not be compromised by a low standard of quality. A web-based environment DigiWeb was created for controlling the workflow of transcribing data from images of natural history specimens. Using DigiWeb, it was possible to manage the workflow of transcription and data proofing, include all participants to the workflow, allow collaboration and training, and also to provide useful processing features. The data emerging from this process pass quality control standards which are supported by DigiWeb and based on the strict requirements of the ISO 2859 standard.

show abstract

“…For example, natural science collections data can often be rife with errors and inconsistencies, and reuse of collections data to address scientific questions imposes concerns of data quality and fitness for use upon the collection's management community. We have continued our development of Kuration 1.0 (Dou et al, 2011), a software package and prototype for automating data curation pipelines with the Kepler scientific workflow system (Ludäscher et al, 2006). Several curation tools and services are integrated into this package as actors, enabling the construction of workflows to perform and document various data curation tasks.…”

Section: Introductionmentioning

confidence: 99%

Towards Automated Design, Analysis and Optimization of Declarative Curation Workflows

Song¹,

Köhler²,

Ludäscher³

et al. 2014

IJDC

View full text Add to dashboard Cite

Data curation is increasingly important. Our previous work on a Kepler curation package has demonstrated advantages that come from automating data curation pipelines by using workflow systems. However, manually designed curation workflows can be error-prone and inefficient due to a lack of user understanding of the workflow system, misuse of actors, or human error. Correcting problematic workflows is often very time-consuming. A more proactive workflow system can help users avoid such pitfalls. For example, static analysis before execution can be used to detect the potential problems in a workflow and help the user to improve workflow design. In this paper, we propose a declarative workflow approach that supports semi-automated workflow design, analysis and optimization. We show how the workflow design engine helps users to construct data curation workflows, how the workflow analysis engine detects different design problems of workflows and how workflows can be optimized by exploiting parallelism.

show abstract

Scientific workflow design 2.0: Demonstrating streaming data collections in Kepler

Cited by 17 publications

References 11 publications

Scientific Workflows and Provenance: Introduction and Research Opportunities

Scientific Workflows and Provenance: Introduction and Research Opportunities

DigiWeb - a workflow environment for quality assurance of transcription in digitization of natural history collections

Towards Automated Design, Analysis and Optimization of Declarative Curation Workflows

Contact Info

Product

Resources

About