Transparent in Situ Data Transformations in ADIOS

Boyuka, David A.; Lakshminarasimhan, Sriram; Zou, Xiaocheng; Gong, Zhenhuan; Jenkins, John; Schendel, Eric R.; Podhorszki, Norbert; Liu, Qing; Klasky, Scott; Samatova, Nagiza F.

doi:10.1109/ccgrid.2014.73

Cited by 25 publications

(12 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Initial work has focused on providing streaming style data transfers between concurrently running simulation and analytics codes, both in-transit [13,11,15] and in situ [39,8,13]. More recent work has focused on providing management capabilities for these mechanisms to address interference when applications share physical resources [38,4,16], as well as resource allocation issues that span entire science workflows [12,16].…”

Section: Related Workmentioning

confidence: 99%

System-Level Support for Composition of Applications

Kocoloski

Lange

Abbasi

et al. 2015

Proceedings of the 5th International Workshop on Runtime and Operating Systems for Supercomputers

View full text Add to dashboard Cite

Current HPC system software lacks support for emerging application deployment scenarios that combine one or more simulations with in situ analytics, sometimes called multi-component or multi-enclave applications. This paper presents an initial design study, implementation, and evaluation of mechanisms supporting composite multi-enclave applications in the Hobbes exascale operating system. These mechanisms include virtualization techniques isolating application custom enclaves while using the vendor-supplied host operating system and high-performance inter-VM communication mechanisms. Our initial single-node performance evaluation of these mechanisms on multi-enclave science applications, both real and proxy, demonstrate the ability to support multi-enclave HPC job composition with minimal performance overhead.

show abstract

Section: Related Workmentioning

confidence: 99%

System-Level Support for Composition of Applications

Kocoloski

Lange

Abbasi

et al. 2015

Proceedings of the 5th International Workshop on Runtime and Operating Systems for Supercomputers

View full text Add to dashboard Cite

show abstract

“…Examples of such frameworks include EPIC [44] and Freeprocessing [46]. A solution based on I/O layer components was proposed in [26]. Remote direct memory access (RDMA) can also contribute by enabling zero-copy, high-throughput and low-latency networking for HPC clusters [38].…”

Section: Framework and Infrastructurementioning

confidence: 99%

State of the Art and Future Trends in Data Reduction for High-Performance Computing

2020

JSFI

View full text Add to dashboard Cite

Research into data reduction techniques has gained popularity in recent years as storage capacity and performance become a growing concern. This survey paper provides an overview of leveraging points found in high-performance computing (HPC) systems and suitable mechanisms to reduce data volumes. We present the underlying theories and their application throughout the HPC stack and also discuss related hardware acceleration and reduction approaches. After introducing relevant use-cases, an overview of modern lossless and lossy compression algorithms and their respective usage at the application and file system layer is given. In anticipation of their increasing relevance for adaptive and in situ approaches, dimensionality reduction techniques are summarized with a focus on non-linear feature extraction. Adaptive approaches and in situ compression algorithms and frameworks follow. The key stages and new opportunities to deduplication are covered next. An unconventional but promising method is recomputation, which is proposed at last. We conclude the survey with an outlook on future developments.

show abstract

“…The fast throughput also leads to wide usage of dictionary‐based coders in I/O middleware and filesystems to provide transparent compression. HDF5 [FCY99] and ADIOS [LKS*08] are two I/O middleware solutions that support transparent compression with zlib among others as plug‐ins [BLZ*14]. Two filesystems, Btrfs [RBM13] and ZFS [BM] on Linux and Solaris operating systems respectively, also natively support transparent compression through dictionary‐based coders such as zlib and gzip [btr, OS12].…”

Section: Truly Lossless Compressionmentioning

confidence: 99%

Data Reduction Techniques for Simulation, Visualization and Data Analysis

Marsaglia

Garth

et al. 2018

Computer Graphics Forum

View full text Add to dashboard Cite

Data reduction is increasingly being applied to scientific data for numerical simulations, scientific visualizations and data analyses. It is most often used to lower I/O and storage costs, and sometimes to lower in‐memory data size as well. With this paper, we consider five categories of data reduction techniques based on their information loss: (1) truly lossless, (2) near lossless, (3) lossy, (4) mesh reduction and (5) derived representations. We then survey available techniques in each of these categories, summarize their properties from a practical point of view and discuss relative merits within a category. We believe, in total, this work will enable simulation scientists and visualization/data analysis scientists to decide which data reduction techniques will be most helpful for their needs.

show abstract

Transparent in Situ Data Transformations in ADIOS

Cited by 25 publications

References 29 publications

System-Level Support for Composition of Applications

System-Level Support for Composition of Applications

State of the Art and Future Trends in Data Reduction for High-Performance Computing

Data Reduction Techniques for Simulation, Visualization and Data Analysis

Contact Info

Product

Resources

About