dispel4py: A Python framework for data-intensive scientific computing

Abstract

This paper presents dispel4py, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. These combine the familiarity of Python programming with the scalability of workflows. Data streaming is used to gain performance, rapid prototyping and applicability to live observations. dispel4py enables scientists to focus on their scientific goals, avoiding distracting details and retaining flexibility over the computing infrastructure they use. The implementation, therefore, has to map dispel4py abstract workflows optimally onto target platforms chosen dynamically. We present four dispel4py mappings: Apache Storm, MPI, multi-threading and sequential, showing two major benefits: a) smooth transitions from local development on a laptop to scalable execution for production work, and b) scalable enactment on significantly different distributed computing infrastructures. Three application domains are reported and measurements on multiple infrastructures show the optimisations achieved; they have provided demanding real applications and helped us develop effective training. The dispel4py.org is an open-source project to which we invite participation. The effective mapping of dispel4py onto multiple target infrastructures demonstrates exploitation of data-intensive and HPC architectures and consistent scalability.

Asterism: Pegasus and Dispel4py Hybrid Workflows for Data-Intensive Science

“…In spite of impressive achievements to date, constructing workflows and orchestrating their executions on heterogeneous systems remain a fundamental challenge. As a result, many workflow management systems (WMSs) have been developed to fulfill specific requirements of different scientific communities [1, 8,13,17,21,22,25,34]. Although these systems have a common goal, they often do not share all capabilities across different e-Infrastructures.…”

“…On the other hand, the growing volumes of scientific data, the increased focus on data-driven science and the achievable storage density doubling every 14 months (Kryder's Law [33]), severely stresses the available disk I/Oor more generally the bandwidth between RAM and external devices. This is driving increased adoption of streambased [17,25] for implementing data-intensive applications, as these avoid a write out to disk followed by reading in, or double that I/O load if files have to be moved. Significantly reducing the cost of data movement between stages makes it economic to compose very simple stages, e.g.…”

“…For example, coarse-grained composition workflow systems, such as Pegasus [8], are mainly mechanisms to organize and distribute computation regardless of the application language, or the target computing infrastructure. Alternatively, parallel dataflow systems (e.g., dispel4py [17]) have the ability to model the movement of data and methods as a series of connection (inputs and outputs), enabling concurrent pipelined and task parallelism, so tasks can execute as soon as the required resources are available. However, unlike coarsegrained WMSs, parallel datalfow systems often do not provide capacities to run workflows across e-Infrastructures.…”

A characterization of workflow management systems for extreme-scale applications

“…• dispel4py [4] implements many of the original Dispel concepts, but presents them as Python constructs. It describes abstract workflows for data-intensive applications, which are later automatically translated to the selected enactment platforms (e.g., Apache Storm, MPI, Multiprocessing, etc.)…”

“…In recent years, numerous workflow management systems (WMSs) have been developed to manage the execution of diverse workflows on heterogeneous computing resources [3,4,5,6,7,8,9]. As user communities adopt and evolve WMSs to fit their own needs, many of the features and capabilities that were once common to most WMSs have become too distinct to share across systems.…”