Data co-processing for extreme scale analysis level II ASC milestone (4745).

Abstract

Exascale supercomputing will embody many revolutionary changes in the hardware and software of high-performance computing. A particularly pressing issue is gaining insight into the science behind the exascale computations. Power and I/O speed constraints will fundamentally change current visualization and analysis workflows. A traditional post-processing workflow involves storing simulation results to disk and later retrieving them for visualization and data analysis. However, at exascale, scientists and analysts will need a range of options for moving data to persistent storage, as the current offline or post-processing pipelines will not be able to capture the data necessary for data analysis of these extreme scale simulations. This Milestone explores two alternate workflows, characterized as in situ and in transit, and compares them. We find each to have its own merits and faults, and we provide information to help pick the best option for a particular use.

Unity

“…New practices associated with realtime experimentation and observational data are driving more complex data sharing between supercomputer simulations and experimental data produced by national science facilities such as acclerators, colliders, and light sources [7]. Projected limitations in the I/O system motivate integrating simulation and analysis, coupling complex physics codes, and developing fully integrated application workflows [9,29,37].…”

Evaluation of methods to integrate analysis into a large-scale shock shock physics code

“…In addition, the large-scale results are more interesting from a supercomputing perspective because they demonstrate capability not possible on smaller clusters. For a complete look at all the results, see our ASC milestone final report [34]. Figure 7 shows the total runtime of each workflow for the large data set.…”

Hobbes

“…The usage model for HPC machines is evolving from sequences of simulation and analysis tasks, communicating via long-term storage, toward a more dynamic, "compositional" approach, where applications consist of complex combinations of coupled codes, data services, and tools. Projected limitations in the I/O system motivate the integration of simulation and analysis, coupling of complex physics codes, and development of fullyintegrated application workflows [35,27,5]. Expected power constraints motivate the need to co-locate applications to avoid data movement wherever possible [27].…”

“…Projected limitations in the I/O system motivate the integration of simulation and analysis, coupling of complex physics codes, and development of fullyintegrated application workflows [35,27,5]. Expected power constraints motivate the need to co-locate applications to avoid data movement wherever possible [27]. New HPC use cases for streaming and graph analytics require features of the OS/R such as global addressing, massive multithreading, and event-based processing that are not well supported on traditional HPC systems [14].…”

“…For example, consider recent work for the National Nuclear Security Administration Advanced Simulation and Computing program (NNSA/ASC) to explore integrated simulation analysis. As part of that project, an SNL team has developed both "in situ" and "in transit" analysis techniques that use ParaView to detect material fragments from simulation results of the CTH shock physics code [27]. This work revealed a number of weaknesses in existing HPC systems that could be dramatically improved by technologies included in the Hobbes design.…”