Leading magnetic fusion energy science into the big-and-fast data lane

Kube, Ralph; Churchill, R.M.; Choi, Jong Youl; Wang, Ruonan; Klasky, Scott; Chang, C.S.; Choi, Minjun; Park, Jinseop

doi:10.25080/majora-342d178e-013

Cited by 4 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We continue by evaluating the performance of computational kernels implementing equation (1). Previous work showed that kernels which implement equation (1) for central processing units (CPUs) can utilize multithreading and scale linearly with the numbers of threads [32]. In this work we utilize a kernel, labeled GAP, that fuses calculations of equations (1a)-(1c) for a given channel pari (X, Y) and runs on GPUs.…”

Section: Individual Componentsmentioning

confidence: 99%

“…In this paper, we present a streaming data analysis framework that aims to connect fusion experiments with remote HPC resources [2,6,19,32]. The streaming paradigm implemented by this framework allows the data generated by fusion experiments to be seamlessly analyzed using supercomputers as they are generated.…”

Section: Introductionmentioning

confidence: 99%

“…As the major contributions of this article, we introduce the aDaptable near rEaL-Time Analysis framework [19,32] that we develop to facilitate streaming analysis of fusion data. Delta targets the rapid execution of arbitrary of analysis code and simulations, using measurements from fusion experiments as input, on remote supercomputers.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Near real-time streaming analysis of big fusion data

Kube¹,

Churchill²,

Chang³

et al. 2022

Plasma Phys. Control. Fusion

View full text Add to dashboard Cite

Experiments on fusion plasmas produce high-dimensional data time series with ever increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article we introduce the DELTA framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, DELTA allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describe the modular and expandable software architecture of DELTA and present performance benchmarks of individual components as well as of an example workflows. Focusing on a streaming analysis workflow where Electron cyclotron emission imaging (ECEi) data measured at KSTAR on NERSC's supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. At NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes in under 5 minutes. We further discuss how DELTA uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. By providing session leaders and physics operators results of higher order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.

show abstract

Section: Individual Componentsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Near real-time streaming analysis of big fusion data

Kube¹,

Churchill²,

Chang³

et al. 2022

Plasma Phys. Control. Fusion

View full text Add to dashboard Cite

show abstract

“…In modern practice, the following situation began to arise. A series of full-scale experiments is being carried out at the facility and in order to make a decision on the next step of the experiment, it is necessary to quickly process the current results [26]. The necessary processing speed can be provided by supercomputer technologies, whereas the program running in such a situation should not be run in batch mode, however it should be run online in order to respond and process data quickly.…”

Section: Systems Analysis Of Visualization Systemsmentioning

confidence: 99%

“…The necessary processing speed can be provided by supercomputer technologies, whereas the program running in such a situation should not be run in batch mode, however it should be run online in order to respond and process data quickly. In practice, it even happens that the facility is located in one part of the planet but the supercomputer is located in another [26].…”

Section: Systems Analysis Of Visualization Systemsmentioning

confidence: 99%

On Expressiveness of Visualization Systems' Interfaces

Vasev¹,

Bakhterev²,

Manakov³

et al. 2022

View full text Add to dashboard Cite

The paper discusses elements of logical models of graphical user interfaces used in both universal and specialized scientific visualization systems. Criteria of expressiveness of programming language that are discussed in "Structure and Interpretation of Computer Programs" book are applied to graphical interfaces. It is shown that graphical interfaces allow users to operate on the same digital substance and with the same logical approaches as in textual programming languages. Both give basic elements, allow their combination, and support the procedures of abstraction. Authors suggest considering these aspects when developing graphical interfaces. This perspective is applied to the following presentation of the paper. The idea of modifiers (also known as behaviors or effects) and the idea of extensions (also known as plugins, modules, and applications) are discussed. Some methods of programming of scene dynamics are presented. Also languages and ontologies of scientific visualization are discussed, e.g. models for editing visualization pipeline: adding data to projects, filtering of that data, and methods of description of data representation on screen. Finally, we discuss additional ideas on systems analysis of visualization systems.

show abstract

mpi4py.futures: MPI-Based Asynchronous Task Execution for Python

Rogowski

Aseeri

Keyes

et al. 2023

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

We present mpi4py.futures, a lightweight, asynchronous task execution framework targeting the Python programming language and using the Message Passing Interface (MPI) for interprocess communication. mpi4py.futures follows the interface of the concurrent.futures package from the Python standard library and can be used as its drop-in replacement, while allowing applications to scale over multiple compute nodes. We discuss the design, implementation, and feature set of mpi4py.futures and compare its performance to other solutions on both shared and distributed memory architectures. On a shared-memory system, we show mpi4py.futures to consistently outperform Python's concurrent.futures with speedup ratios between 1.4X and 3.7X in throughput (tasks per second) and between 1.9X and 2.9X in bandwidth. On a Cray XC40 system, we compare mpi4py.futures to Dask -a well-known Python parallel computing package. Although we note more varied results, we show mpi4py.futures to outperform Dask in most scenarios.

show abstract

Leading magnetic fusion energy science into the big-and-fast data lane

Cited by 4 publications

References 13 publications

Near real-time streaming analysis of big fusion data

Near real-time streaming analysis of big fusion data

On Expressiveness of Visualization Systems' Interfaces

mpi4py.futures: MPI-Based Asynchronous Task Execution for Python

Contact Info

Product

Resources

About