Parallel I/O, analysis, and visualization of a trillion particle simulation

Byna, Suren; Chou, Jerry; Rübel, Oliver; Prabhat,; Karimabadi, H.; Daughter, William S.; Roytershteyn, V.; Bethel, E. Wes; Howison, Mark; Hsu, Ke-Jou; Lin, Kuan-Wu; Shoshani, Arie; Uselton, Andrew; Wu, Kesheng

doi:10.1109/sc.2012.92

Cited by 71 publications

(55 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Individual computational particles represent many physical particles (for example, electrons). The particular dataset we used in our experiments is a 1024 3 random subsample of the trillion-particle dataset used in a parallel I/O study by Byna et al [35]. Delaunay and Voronoi tessellations can help describe the spatial distribution of highly energetic particles (including identifying their dense regions), although in this paper we use this dataset to further measure the scalability of our algorithm.…”

Section: Soft Matter Simulationsmentioning

confidence: 99%

High-Performance Computation of Distributed-Memory Parallel 3D Voronoi and Delaunay Tessellation

Peterka

Morozov

Phillips

2014

SC14: International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

ABSTRACT. Computing a Voronoi or Delaunay tessellation from a set of points is a core part of the analysis of many simulated and measured datasets: N-body simulations, molecular dynamics codes, and LIDAR point clouds are just a few examples. Such computational geometry methods are common in data analysis and visualization; but as the scale of simulations and observations surpasses billions of particles, the existing serial and shared-memory algorithms no longer suffice. A distributed-memory scalable parallel algorithm is the only feasible approach. The primary contribution of this paper is a new parallel Delaunay and Voronoi tessellation algorithm that automatically determines which neighbor points need to be exchanged among the subdomains of a spatial decomposition. Other contributions include periodic and wall boundary conditions, comparison of our method using two popular serial libraries, and application to numerous science datasets.E-mail addresses: tpeterka@mcs.anl.gov, dmitriy@mrzv.org, cphillips@anl.gov. DISCLAIMER. This document was prepared as an account of work sponsored by the United States Government. While this document is believed to contain correct information, neither the United States Government nor any agency thereof, nor the Regents of the University of California, nor any of their employees, makes any warranty, express or implied, or assumes any legal responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by its trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof, or the Regents of the University of California. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof or the Regents of the University of California.

show abstract

Section: Soft Matter Simulationsmentioning

confidence: 99%

High-Performance Computation of Distributed-Memory Parallel 3D Voronoi and Delaunay Tessellation

Peterka

Morozov

Phillips

2014

SC14: International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

show abstract

“…Recent work by Byna et al 21 has approached this daunting set of scientific questions in a holistic way that considers the end-to-end problem of producing, storing, analyzing, and visualizing plasma physics simulation data of unprecedented scale. Given the science questions above and the desire to run the VPIC simulation at very high resolution, Byna et al addressed data management challenges in storing, analyzing, and visualizing simulation output.…”

Section: Plasma Physics Data Analyticsmentioning

confidence: 99%

“…Byna et al conducted a series of weak-scaling experiments designed to study the behavior of the collective, parallel I/O approach and to compare it with the traditional fpp approach. Their results, described in more detail the original study, 21 show that their approach is able to achieve a sustained 77% of peak I/O rate at 120K cores (≈27GB/s measured, ≈35GB/s theoretical peak). The fpp approach initially achieved a larger percent of peak performance, but then the performance falls off due to load imbalance in the two-stage non-collective I/O process.…”

mentioning

confidence: 94%

Why high performance visual data analytics is both relevant and difficult

et al. 2013

Self Cite

View full text Add to dashboard Cite

Data visualization, as well as data analysis and data analytics, are all an integral part of the scientific process. Collectively, these technologies provide the means to gain insight into data of ever-increasing size and complexity. Over the past two decades, a substantial amount of visualization, analysis, and analytics R&D has focused on the challenges posed by increasing data size and complexity, as well as on the increasing complexity of a rapidly changing computational platform landscape. While some of this research focuses on solely on technologies, such as indexing and searching or novel analysis or visualization algorithms, other R&D projects focus on applying technological advances to specific application problems. Some of the most interesting and productive results occur when these two activities-R&D and application-are conducted in a collaborative fashion, where application needs drive R&D, and R&D results are immediately applicable to real-world problems.

show abstract

“…We illustrate the utility of our approach by applying it to two science simulation use cases: VPIC-I/O [5], [4] and HACC-I/O [12], [2]. Specifically, this paper makes the following contributions: 1) Introduction of an object representation for mapping scientific data into object abstractions that stay uniform across storage hierachies.…”

Section: Introductionmentioning

confidence: 99%

Toward Transparent Data Management in Multi-Layer Storage Hierarchy of HPC Systems

Wadhwa

Byna

Butt

2018

2018 IEEE International Conference on Cloud Engineering (IC2E)

View full text Add to dashboard Cite

Abstract-Upcoming exascale high performance computing (HPC) systems are expected to comprise multi-tier storage hierarchy, and thus will necessitate innovative storage and I/O mechanisms. Traditional disk and block-based interfaces and file systems face severe challenges in utilizing capabilities of storage hierarchies due to the lack of hierarchy support and semantic interfaces. Object-based and semantically-rich data abstractions for scientific data management on large scale systems offer a sustainable solution to these challenges. Such data abstractions can also simplify users involvement in data movement. In this paper, we take the first steps of realizing such an object abstraction and explore storage mechanisms for these objects to enhance I/O performance, especially for scientific applications. We explore how an object-based interface can facilitate next generation scalable computing systems by presenting the mapping of data I/O from two real world HPC scientific use cases: a plasma physics simulation code (VPIC) and a cosmology simulation code (HACC). Our storage model stores data objects in different physical organizations to support data movement across layers of memory/storage hierarchy. Our implementation sclaes well to 16K parallel processes, and compared to the state of the art, such as MPI-IO and HDF5, our object-based data abstractions and data placement strategy in multi-level storage hierarchy achieves up to 7× I/O performance improvement for scientific data.

show abstract

Parallel I/O, analysis, and visualization of a trillion particle simulation

Cited by 71 publications

References 34 publications

High-Performance Computation of Distributed-Memory Parallel 3D Voronoi and Delaunay Tessellation

High-Performance Computation of Distributed-Memory Parallel 3D Voronoi and Delaunay Tessellation

Why high performance visual data analytics is both relevant and difficult

Toward Transparent Data Management in Multi-Layer Storage Hierarchy of HPC Systems

Contact Info

Product

Resources

About