Collection-Oriented Scientific Workflows for Integrating and Analyzing Biological Data

Abstract

Abstract. Steps in scientific workflows often generate collections of results, causing the data flowing through workflows to become increasingly nested. Because conventional workflow components (or actors) typically operate on simple or application-specific data types, additional actors often are required to manage these nested data collections. As a result, conventional workflows become increasingly complex as data becomes more nested. This paper describes a new paradigm for developing scientific workflows that transparently manages nested data collections. Collection-oriented workflows have a number of advantages over conventional approaches including simpler workflow designs (e.g., requiring fewer actors and control-flow constructs) that are invariant under changes in data nesting. Our implementation within the Kepler scientific workflow system enables the explicit representation of collections and collection schemas, concurrent operation over collection contents via multi-level pipeline parallelism, and allows collection-aware actors to be composed readily from conventional actors.

Querying an Integrated Complex-Object Dataflow Database

“…Support for the complex-object structure of data flowing in a scientific workflow is present in various systems, e.g., Taverna [10,11], Kepler/CoMaD [12,13], and Chimera [14,15]. The operation of applying a function on all elements of a collection is typically provided.…”

Linking multiple workflow provenance traces for interoperable collaborative science

“…Both sub-workflows were encoded in Kepler and Taverna, and combined in a "crossing-over" manner in several ways. Overall, three different models of computation (MoCs) and corresponding designs were used for each of W PC1 and W PC2 : Taverna, "conventional" Kepler [4], and Kepler/COMAD [20], a novel MoC with its own provenance recorder to handle fine-grained dependencies within nested data collections [9], [6]. In the implementation, we have used a combination of native provenance models, available from both workflow systems, as well as OPM provenance graphs derived from those models (Fig.…”

Comparative Studies Simplified in GPFlow

“…To that end, we first review models of data flow programming that relate to collection processing. We subsequently examine collection processing in two extant scientific workflow systems, namely Kepler [4], [5] and Taverna [6].…”