Workflow Skeletons: Increasing Scalability of Scientific Workflows by Combining Orchestration and Choreography

Abstract

Dataflow modeling is the natural way of composing scientific workflows, because they often comprise numerous data transformation steps applying massive parallelism. However, modeling control flow within dataflow is often achieved at the expense of clarity and comprehensibility.This paper describes scientific workflows maintaining the robustness of centralized control (using orchestration) by modeling control flow, while at the same time integrating sub-workflows that are modeled by workflow skeletons (using choreography) describing dataflow.Following the concept of algorithmic skeletons, we define workflow skeletons as re-usable parallel constructs describing dataflow connections between proxies representing services. Proxies are able to communicate with each other allowing for efficient coupling between parallel tasks and avoiding of unnecessary data transfers. Skeletons increase scalability on demand by accepting the number of parallel tasks.The primary contributions are a formal model describing workflow skeletons and a script language "Workflow Skeleton Language" (WorkSKEL). Furthermore, this paper demonstrates the definition of selected workflow patterns like pipeline and farm in WorkSKEL.

On the Support of Scientific Workflows over Pub/Sub Brokers

“…The high degree of dynamism inherent to these systems is not easily modeled or scaled [17] by a business WfMS, which provides orchestration with a centralized scheduling environments which also usually implement centralized messages in the control plane. On the other hand, a simple choreography approach cannot be used as it is difficult to keep track of all the task instances and workflow activities at any given time [18]. Hence, in order to support the previous characteristics, we use a Pub/Sub model for delivering these control messages in the whole SWf execution.…”

“…Some approaches propose decentralized SWfMS which optimize communication by placing each orchestration engine as close as possible to the component service it manages [18], so, our work targets these kinds of systems.…”

Facilitating Scientific Workflow Configuration with Parameterized Workflow Skeletons

“…Based on a formal model of the concept of workflow Skeleton we define a specification language to specify workflow Skeletons: Skeleton Workflow Language (WorkSKEL) [6].…”

“…In our former work, we introduced the concept workflow skeletons for describing parallel patterns [6]; they are reusable parallel building blocks that scientists use as black boxes without needing to know their internal structure. Workflow Skeletons are parametrizable, re-usable, and nested sub-workflows that can be defined in our script language WorkSKEL.…”

A Life Cycle for Coupled Multi-scale, Multi-field Experiments Realized through Choreographies

“…Fleuren et al [23] propose an approach and an implementation for integrating control and data flow by combining orchestration and choreography. The main workflow is modeled as an orchestration using a control flow perspective also considering fault handling and compensation.…”