ADR visualization: A generalized framework for ranking large-scale scientific data using Analysis-Driven Refinement

Nouanesengsy, Boonthanome; Woodring, Jonathan; Patchett, John; Myers, Kary; Ahrens, James

doi:10.1109/ldav.2014.7013203

Cited by 37 publications

(30 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An alternative approach would be to perform expensive analyses and I/O in an adaptive fashion, driven by the data itself. In [19,18], such techniques have been developed based on entropy of information in the data, and building piecewise-linear fits of quantities of interest. These approaches fit within the methodology proposed here and are domain-agnostic.…”

Section: 2)mentioning

confidence: 99%

Trigger Detection for Adaptive Scientific Workflows Using Percentile Sampling

Bennett¹,

Bhagatwala²,

Chen³

et al. 2016

SIAM J. Sci. Comput.

View full text Add to dashboard Cite

Abstract. Increasing complexity of both scientific simulations and high performance computing system architectures are driving the need for adaptive workflows, in which the composition and execution of computational and data manipulation steps dynamically depend on the evolutionary state of the simulation itself. Consider for example, the frequency of data storage. Critical phases of the simulation should be captured with high frequency and with high fidelity for post-analysis, however we cannot afford to retain the same frequency for the full simulation due to the high cost of data movement. We can instead look for triggers, indicators that the simulation will be entering a critical phase, and adapt the workflow accordingly.In this paper, we present a methodology for detecting triggers and demonstrate its use in the context of direct numerical simulations of turbulent combustion using S3D. We show that chemical explosive mode analysis (CEMA) can be used to devise a noise-tolerant indicator for rapid increase in heat release. However, exhaustive computation of CEMA values dominates the total simulation, thus is prohibitively expensive. To overcome this computational bottleneck, we propose a quantile sampling approach. Our sampling based algorithm comes with provable error/confidence bounds, as a function of the number of samples. Most importantly, the number of samples is independent of the problem size, thus our proposed sampling algorithm offers perfect scalability. Our experiments on homogeneous charge compression ignition (HCCI) and reactivity controlled compression ignition (RCCI) simulations show that the proposed method can detect rapid increases in heat release, and its computational overhead is negligible. Our results will be used to make dynamic workflow decisions regarding data storage and mesh resolution in future combustion simulations. The proposed sampling-based framework is generalizable and we detail how it could be applied to a broad class of scientific simulation workflows.Keywords: Sublinear algorithms; quantile sampling; in situ data analysis; chemical explosive mode analysis (CEMA); S3D; adaptive workflow; judicious I/O; 1. Introduction. Steady improvements in computing resources enable ever more enhanced scientific simulations, however Input/Output (I/O) constraints are impeding their impact. Historically, scientific computing workflows have been defined by three independent stages (see Fig. 1.1(a)): 1) a pre-processing stage comprising initialization and set up (for example mesh generation, or initial small-scale test runs); 2) the scientific computation itself (in which data is periodically saved to disk at a prescribed frequency); and 3) post-processing and analysis of data for scientific insights. With improved computing resources scientists are increasing the temporal resolution of their simulations. However, as computational power continues to outpace I/O capabilities, the gap between time steps saved to disk keeps increasing. This compromise in the fidelity of the data being saved t...

show abstract

Section: 2)mentioning

confidence: 99%

Trigger Detection for Adaptive Scientific Workflows Using Percentile Sampling

Bennett¹,

Bhagatwala²,

Chen³

et al. 2016

SIAM J. Sci. Comput.

View full text Add to dashboard Cite

show abstract

“…First, Lehmann et al, 2014 [LJ14] consider both multiresolution and temporal compression of data using a technique that can generate adaptively refined meshes. Nouanesengsy et al, 2014 [NWP*14] introduced a scheme around Analysis‐Driven Refinement (ADR), also inspired by the principles behind adaptive mesh refinement. Fernandes et al, 2014 [FFSE14] extend volumetric depth images (VDI) for compression of simulation results for post hoc visualization.…”

Section: In Situ History and Surveymentioning

confidence: 99%

In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms

Bauer

Abbasi

Ahrens

et al. 2016

Computer Graphics Forum

147

View full text Add to dashboard Cite

The considerable interest in the high performance computing (HPC) community regarding analyzing and visualization data without first writing to disk, i.e., in situ processing, is due to several factors. First is an I/O cost savings, where data is analyzed/visualized while being generated, without first storing to a filesystem. Second is the potential for increased accuracy, where fine temporal sampling of transient analysis might expose some complex behavior missed in coarse temporal sampling. Third is the ability to use all available resources, CPU's and accelerators, in the computation of analysis products. This STAR paper brings together researchers, developers and practitioners using in situ methods in extreme-scale HPC with the goal to present existing methods, infrastructures, and a range of computational science and engineering applications using in situ analysis and visualization.

show abstract

“…Nouanesengsy et al [14] have also proposed a framework to filter out unimportant data prior to in situ visualization. Their approach is based on analysis-driven refinement, which partitions datasets according to a user-defined importance criteria.…”

Section: A Adaptive In Situ Visualizationmentioning

confidence: 99%

Adaptive Performance-Constrained In Situ Visualization of Atmospheric Simulations

Dorier

Sisneros²,

Bautista-Gomez

et al. 2016

2016 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

While many parallel visualization tools now provide in situ visualization capabilities, the trend has been to feed such tools with large amounts of unprocessed output data and let them render everything at the highest possible resolution. This leads to an increased run time of simulations that still have to complete within a fixed-length job allocation. In this paper, we tackle the challenge of enabling in situ visualization under performance constraints. Our approach shuffles data across processes according to its content and filters out part of it in order to feed a visualization pipeline with only a reorganized subset of the data produced by the simulation. Our framework leverages fast, generic evaluation procedures to score blocks of data, using information theory, statistics, and linear algebra. It monitors its own performance and adapts dynamically to achieve appropriate visual fidelity within predefined performance constraints. Experiments on the Blue Waters supercomputer with the CM1 simulation show that our approach enables a 5× speedup with respect to the initial visualization pipeline and is able to meet performance constraints.

show abstract

ADR visualization: A generalized framework for ranking large-scale scientific data using Analysis-Driven Refinement

Cited by 37 publications

References 26 publications

Trigger Detection for Adaptive Scientific Workflows Using Percentile Sampling

Trigger Detection for Adaptive Scientific Workflows Using Percentile Sampling

In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms

Adaptive Performance-Constrained In Situ Visualization of Atmospheric Simulations

Contact Info

Product

Resources

About