Pygion: Flexible, Scalable Task-Based Parallelism with Python

Slaughter, Elliott; Aiken, Alex

doi:10.1109/paw-atm49560.2019.00011

Cited by 14 publications

(10 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We originally anticipated that FLEET applications would be written in Regent, 18 a language that supports implicit dataflow parallelism, or in C++. However, as the project evolved we shifted to Pygion, 17 The example of 80 Tbps is not a maximum limit: using more servers in the local cluster or across the WAN to other clusters could increase this simultaneous in-flight bandwidth usage arbitrarily.…”

Section: Software Overviewmentioning

confidence: 99%

FLEET—Fast Lanes for Expedited Execution at 10 Terabits: Program Overview

Douglis¹,

Robertson²,

Berg³

et al. 2021

IEEE Internet Comput.

Self Cite

View full text Add to dashboard Cite

Section: Software Overviewmentioning

confidence: 99%

FLEET—Fast Lanes for Expedited Execution at 10 Terabits: Program Overview

Douglis¹,

Robertson²,

Berg³

et al. 2021

IEEE Internet Comput.

Self Cite

View full text Add to dashboard Cite

“…First, the default execution space is set (line 35). Second, the Views y, x, and A are created by calling the View() constructor (lines [36][37][38]. The first argument to the constructor is a list of the View's dimensions.…”

Section: Pykokkos Via An Examplementioning

confidence: 99%

“…Unlike prior work, PyKokkos enables performance portability across HPC frameworks by targeting the C++ Kokkos library and supports the latest version of Python. Dask [35] and Pygion [38] enable distributed task-based programming in Python. PyKokkos focuses on shared-memory parallelism instead.…”

Section: Related Workmentioning

confidence: 99%

A performance portability framework for Python

Awar

Zhu

Biros

et al. 2021

Proceedings of the ACM International Conference on Supercomputing

View full text Add to dashboard Cite

Kokkos is a programming model for writing performance portable applications for all major high performance computing platforms. It provides abstractions for data management and common parallel operations, allowing developers to write portable high performance code with minimal knowledge of architecture-specific details. Kokkos is implemented as a heavily-templated C++ library. However, C++ is not ideal for rapid prototyping and quick algorithmic exploration. An increasing number of developers use Python for scientific computing, machine learning, and data analytics. In this paper, we present a new Python framework, dubbed PyKokkos, for writing performance portable applications entirely in Python. PyKokkos provides Kokkos-like abstractions that are easier to use and more concise than the C++ interface. We implemented PyKokkos by building a translator from a subset of Python to C++ Kokkos and bridging necessary function calls via automatically generated Python bindings. PyKokkos is also compatible with NumPy, a widely-used high performance Python library. By porting several existing Kokkos applications to PyKokkos, including ExaMiniMD (∼3k lines of code in C++), we show that the latter can achieve efficient execution with low performance overhead. CCS CONCEPTS• Software and its engineering → Source code generation; • Computing methodologies → Parallel programming languages.

show abstract

“…To parallelize and scale the application we use Pygion, a Python interface for the Legion task-based programming system [SA19]. In Pygion, the user decorates functions as tasks, and annotates task parameters with privileges (read, write, reduce), but otherwise need not be concerned with how tasks execute on the underlying machine.…”

Section: Parallelization: Effortless Scaling With Pygionmentioning

confidence: 99%

Fluctuation X-ray Scattering real-time app

Dujardin¹,

Slaugther²,

Donatelli³

et al. 2020

Proceedings of the Python in Science Conference

View full text Add to dashboard Cite

The Linac Coherent Light Source (LCLS) at the SLAC National Accelerator Laboratory is an X-ray Free Electron Laser (X-FEL) facility enabling scientists to take snapshots of single macromolecules to study their structure and dynamics. A major LCLS upgrade, LCLS-II, will bring the repetition rate of the X-ray source from 120 to 1 million pulses per second and exascale High Performance Computing (HPC) capabilities will be required for the data analysis to keep up with the future data taking rates. We present here a Python application for Fluctuation X-ray Scattering (FXS), an emerging technique for analyzing biomolecular structure from the angular correlations of FEL diffraction snapshots with one or more particles in the beam. This FXS application for experimental data analysis is being developed to run on supercomputers in near real-time while an experiment is taking place. We discuss how we accelerated the most compute intensive parts of the application and how we used Pygion, a Python interface for the Legion taskbased programming model, to parallelize and scale the application.

show abstract

Pygion: Flexible, Scalable Task-Based Parallelism with Python

Cited by 14 publications

References 22 publications

FLEET—Fast Lanes for Expedited Execution at 10 Terabits: Program Overview

FLEET—Fast Lanes for Expedited Execution at 10 Terabits: Program Overview

A performance portability framework for Python

Fluctuation X-ray Scattering real-time app

Contact Info

Product

Resources

About