PyCOMPSs: Parallel computational workflows in Python

Tejedor, Enric; Becerra, Yolanda; Alomar, Guillem; Queralt, Anna; Badía, Rosa M.; Torres, Jordi; Cortes, Toni; Labarta, Jesús

doi:10.1177/1094342015594678

Cited by 125 publications

(114 citation statements)

References 11 publications

Supporting

Mentioning

111

Contrasting

Unclassified

Order By: Relevance

“…PyCOMPSs [38], a Python interface around the COMPSs system, is a framework for parallel computing in Python. As with Parsl, users decorate functions with constructs to aid workflow assembly and execution.…”

Section: Related Workmentioning

confidence: 99%

Parsl

Babuji

Woodard

et al. 2019

Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing

174

View full text Add to dashboard Cite

High-level programming languages such as Python are increasingly used to provide intuitive interfaces to libraries written in lower-level languages and for assembling applications from various components. This migration towards orchestration rather than implementation, coupled with the growing need for parallel computing (e.g., due to big data and the end of Moore's law), necessitates rethinking how parallelism is expressed in programs. Here, we present Parsl, a parallel scripting library that augments Python with simple, scalable, and flexible constructs for encoding parallelism. These constructs allow Parsl to construct a dynamic dependency graph of components that it can then execute efficiently on one or many processors. Parsl is designed for scalability, with an extensible set of executors tailored to different use cases, such as low-latency, high-throughput, or extreme-scale execution. We show, via experiments on the Blue Waters supercomputer, that Parsl executors can allow Python scripts to execute components with as little as 5 ms of overhead, scale to more than 250 000 workers across more than 8000 nodes, and process upward of 1200 tasks per second. Other Parsl features simplify the construction and execution of composite programs by supporting elastic provisioning and scaling of infrastructure, fault-tolerant execution, and integrated wide-area data management. We show that these capabilities satisfy the needs of many-task, interactive, online, and machine learning applications in fields such as biology, cosmology, and materials science.

show abstract

Section: Related Workmentioning

confidence: 99%

Parsl

Babuji

Woodard

et al. 2019

Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing

174

View full text Add to dashboard Cite

show abstract

“…[2][3][4][5] Task-based workflows are task-parallel, high-throughput workflows that exchange data through files, can run across several independent systems in a wide area such as grids and clouds but do not have to. [6][7][8][9] Heterogeneous workflow systems such as in situ and taskbased ones lack interoperability with each other. Hence, today's workflow solutions often cannot support an automated end-to-end path from hypothesis to discovery.…”

Section: Heterogeneous Hierarchical Workflow Compositionmentioning

confidence: 99%

Heterogeneous Hierarchical Workflow Composition

et al. 2019

View full text Add to dashboard Cite

Workflow systems promise scientists an automated end-to-end path from hypothesis to discovery. However, expecting any single workflow system to deliver such a wide range of capabilities is impractical. A more practical solution is to compose the end-to-end workflow from more than one system. With this goal in mind, the integration of task-based and in situ workflows is explored, where the result is a hierarchical heterogeneous workflow composed of subworkflows, with different levels of the hierarchy using different programming, execution, and data models. Materials science use cases demonstrate the advantages of such heterogeneous hierarchical workflow composition.Scientific computing consists of multiple related computational tasks. For instance, the detection of highly turbulent potentially destructive features in plasma physics simulations requires the in situ coupling of two simulation codes, a compression tool, a feature detection algorithm, and a visualization library. 1 Such complex workflows require significant effort from scientists to manage the scheduling and data exchange among those tasks. To help automate this process, scientific workflow frameworks allow scientists to define the dependencies and data exchanges among connected tasks instead of managing those manually, potentially resulting in increased scientific productivity.

show abstract

“…COMPSs [32,53] is a framework which aims to ease the development and execution of parallel applications for distributed infrastructures, such as Clusters and Clouds. A COMPSs application is composed of tasks, which are annotated methods.…”

Section: Implementing Two-level Workflows For Astronomy With Taverna mentioning

confidence: 99%

“…Programmatic interfaces does not only support the description of iterative constructions like conditional loops, but offer the whole expressiveness of the programming language to express complex algorithms, like optimization searches, etc. For example, PyCOMPSs [53] or Swift [55] offer programmatic/scripting interfaces.Additionally, taking into account that some application areas may require the possibility of accepting streamed input data (from sensors or other sources of dynamic data) and streamed output data (visualization, monitoring, etc) the system should support this type of data acquisition.This first coarser grain level of workflows will include a set of analytics, implemented as fine grain workflows. These analytics can be provided as a layer of Analytics as a Service that can be used by the workflows depending on their requirements.…”

mentioning

confidence: 99%

Workflows for science: a challenge when facing the convergence of HPC and Big Data

Badía

Ayguadé

Labarta

2017

JSFI

View full text Add to dashboard Cite

Workflows have been traditionally a mean to describe and implement the computing experiments, usually parametric studies and explorations searching for the best solution, that scientific researchers want to perform. A workflow is not only the computing application, but a way of documenting a process. Science workflows may be of very different nature depending on the area of research, matching the actual experiment that the scientist want to perform. Workflow Management Systems are environments that offer the researchers tools to define, publish, execute and document their workflows.In some cases, the science workflows are used to generate data; in other cases are used to analyse existing data; only in a few cases, workflows are used both to generate and analyse data. The design of experiments is in some cases generated blindly, without a clear idea of which points are relevant to be computed/simulated, ending up with huge amount of computation that is performed following a brute-force strategy.However, the evolution of systems and the large amount of data generated by the applications require an in-situ analysis of the data, thus requiring new solutions to develop workflows that includes both the simulation/computational part and the analytic part. What is more, the fact that both components, computation and analytics, can be run together will enable the possibility of defining more dynamic workflows, with new computations being decided by the analytics in a more efficient way.The first part of the paper will review current approaches that a set of scientific communities follow in the development of their workflows. This paper does not intent to be exhaustive in the compilation of different approaches available to develop and deploy workflows. We focus on the Workflow Management Systems used by a set of scientific communities and their representative use cases, with the objective of understanding their different needs and requirements. The second part of the paper proposes a new software architecture to develop a new family of end-to-end workflows that enables the management of dynamic workflows composed of simulations, analytics and visualization, including inputs/outputs from streams.Keywords: workflows, scientific applications, Big Data. IntroductionWorkflows appeared last century and have been used in the manufacturing industry as a mean to optimize their processes. Examples of traditional (non-IT) workflows can be found in the assembly lines, i.e., the Ford Model T assembly line standardized the production processes and was the first continuous delivery pipeline for the automotive industry. This process reduced the costs of manufacturing from $850 to $260 in 1924.The time and motion studies defined by Taylor [52] and Gilbreth [41] had significant impact in the manufacturing processes. These studies proposed to break manufacturing activities into small, simple steps, to determine with accuracy the amount of time required to perform each of the steps. Then, the sequence of movements taken by the employee has...

show abstract

PyCOMPSs: Parallel computational workflows in Python

Cited by 125 publications

References 11 publications

Parsl

Parsl

Heterogeneous Hierarchical Workflow Composition

Workflows for science: a challenge when facing the convergence of HPC and Big Data

Contact Info

Product

Resources

About