Raw data queries during data-intensive parallel workflow execution

Abstract

Computer simulations consume and produce huge amounts of raw data files presented in different formats, e.g., HDF5 in computational fluid dynamics simulations. Users often need to analyze domain-specific data based on related data elements from multiple files during the execution of computer simulations. In a raw data analysis, one should identify regions of interest in the data space and retrieve the content of specific related raw data files. Existing solutions, such as FastBit and RAW, are limited to a single raw data file analysis and can only be used after the execution of computer simulations. Scientific Workflow Management Systems (SWMS) can manage the dataflow of computer simulations and register related raw data files at a provenance database. This paper aims to combine the advantages of a dataflow-aware SWMS and the raw data file analysis techniques to allow for queries on raw data file elements that are related, but reside in separate files. We propose a component-based architecture, named as ARMFUL (Analysis of Raw data from Multiple Files) with raw data extraction and indexing techniques, which allows for a direct access to specific elements or regions of raw data space. ARMFUL innovates by using a SWMS provenance database to add a dataflow access path to raw data files. ARMFUL facilitates the invocation of ad-hoc programs and third party tools (e.g., FastBit tool) for raw data analyses. In our experiments, a real parallel computational fluid dynamics is executed, exploring different alternatives of raw data extraction, indexing and analysis.

Data reduction in scientific workflows using provenance monitoring and user steering

“…We use paths that point to these data files on disk in the attributes of the data elements composing the datasets 1 ∈ whenever a file is read or written by an activity [7]. Some other domain-specific data values within those files can be extracted or generated based on the content of the file and represented as attributes in the data elements, also increasing the expressivity of the file descriptions [13,14]. These domain-specific quantities can be decisive for the domain so they need to be tracked.…”

“…Data management in scientific workflows is critical due to the inherent complexity of the scientific domain data and the HPC requirements, such as efficient exploitation of data parallelism. In this section, we provide the background for this work, which relies on a data-centric algebraic approach for scientific workflows [13,14]. It provides constructs, mechanisms, and conceptualizations which in essence aim at valorizing fine-grained elements of data flowing throughout the workflow activities, rather than just the chaining of tasks (i.e., chaining of programs or processes).…”

Active Provenance for Data-Intensive Workflows: Engaging Users and Developers

“…Many WMSs adopt a database to store lineage data [24]. Other work emphasises queries over the stored data, while these are generated [30]. We present details about storage and access of lineage information in a separate paper.…”

Scientific workflows: Past, present and future

“…This section is sub-divided into three complementary areas related to optimising workflows performance: parallel computing [24], scheduling and planning [25,26,27,28], and data management [29,30].…”