Heterogeneous Hierarchical Workflow Composition

Abstract

Workflow systems promise scientists an automated end-to-end path from hypothesis to discovery. However, expecting any single workflow system to deliver such a wide range of capabilities is impractical. A more practical solution is to compose the end-to-end workflow from more than one system. With this goal in mind, the integration of task-based and in situ workflows is explored, where the result is a hierarchical heterogeneous workflow composed of subworkflows, with different levels of the hierarchy using different programming, execution, and data models. Materials science use cases demonstrate the advantages of such heterogeneous hierarchical workflow composition.Scientific computing consists of multiple related computational tasks. For instance, the detection of highly turbulent potentially destructive features in plasma physics simulations requires the in situ coupling of two simulation codes, a compression tool, a feature detection algorithm, and a visualization library. 1 Such complex workflows require significant effort from scientists to manage the scheduling and data exchange among those tasks. To help automate this process, scientific workflow frameworks allow scientists to define the dependencies and data exchanges among connected tasks instead of managing those manually, potentially resulting in increased scientific productivity.

A novel transversal processing model to build environmental big data services in the cloud

“…The main usage of the solutions based on a hierarchical approach [57] is the discovery of PSs and the composition of different solutions as a single one. It is usually based on two hierarchy levels.…”

X-composer

“…Beeflow [9] is a workflow management system that supports traditional workflows as well as workflows with in situ analysis, and it utilizes events-synchronization primitives to enforce in situ workflow logic. Yildiz et al [49] explore the combination of task-based computing model and in situ workflows using Decaf and PyCOMPs, by integrating Decaf's in situ components as a sub-workflow of the task-based PyCOMPs workflow. Badia et al [5] use a holistic way to express a workflow, where both data/computing are integrated into a single flow built on high-level interfaces.…”

Priority research directions for in situ data management: Enabling scientific discovery from diverse data sources

“…A dichotomy remains between in situ and distributed workflows despite efforts to integrate these communities (Deelman et al, 2016). Even though heterogeneous hierarchical combinations of multiple workflow models are starting to appear in the research literature (Yildiz et al, 2019), hybrid workflows in practice are limited to single use cases. CyberShake, a seismic hazard model from the Southern California Earthquake Center (Graves et al, 2011), combines high-performance and high-throughput computing.…”

“…Real-time experimental and observational data analysis and assimilation of streaming, potentially noisy, and time-critical data. As a concrete example, the workflow in Figure 2 illustrates some of the above elements: an ensemble of numerous instances of molecular dynamics simulations is launched and analyzed, searching for the signature of the stochastic process of nucleation as a material cools and crystallizes (Yildiz et al, 2019). Crystal structures are detected in situ, and only features of interest are saved to storage.…”