Supporting XML Based High-Level Abstractions on HDF5 Datasets: A Case Study in Automatic Data Virtualization

Sahoo, Sanjaya Kumar; Agrawal, Gagan

doi:10.1007/11532378_22

Cited by 3 publications

(2 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sahoo and Agrawal propose a solution for the problem of automatically creating efficient data manipulation services for high-level visualization of low-level data [19]. This solution employs a mapping from HDF5 into XML schema in order to achieve a way to manipulate low-level data through functions using XQuery.…”

Section: Related Workmentioning

confidence: 99%

Optimization of PACS Data Persistency Using Indexed Hierarchical Data

Prado¹,

Macedo

Dantas³

et al. 2014

J Digit Imaging

View full text Add to dashboard Cite

We present a new approach for the development of a data persistency layer for a Digital Imaging and Communications in Medicine (DICOM)-compliant Picture Archiving and Communications Systems employing a hierarchical database. Our approach makes use of the HDF5 hierarchical data storage standard for scientific data and overcomes limitations of hierarchical databases employing inverted indexing for secondary key management and for efficient and flexible access to data through secondary keys. This inverted indexing is achieved through a general purpose document indexing tool called Lucene. This approach was implemented and tested using real-world data against a traditional solution employing a relational database, in various store, search, and retrieval experiments performed repeatedly with different sizes of DICOM datasets. Results show that our approach outperforms the traditional solution on most of the situations, being more than 600 % faster in some cases.

show abstract

Section: Related Workmentioning

confidence: 99%

Optimization of PACS Data Persistency Using Indexed Hierarchical Data

Prado¹,

Macedo

Dantas³

et al. 2014

J Digit Imaging

View full text Add to dashboard Cite

show abstract

“…In the file I/O model, science applications exchange data by using a shared filesystem as the data exchange medium [27], and workflows are constructed through the use of intermediate files stored on disk. As a familiar scientific scenario, consider the following: at each output epoch the parallel writers open a file, encode their data in the proper metadatarich serialization format, like NetCDF [28] or HDF5 [29], populate the write buffers, and finally, flush them to disk. Similarly, for a given read epoch the (possibly parallel) readers open the file, read metadata about the objects present in the file, create the appropriate buffers, perform the reads, and then seek ahead to the next block of data, if available.…”

Section: Type-based Publish Subscribe For Hpcmentioning

confidence: 99%

Flexpath: Type-Based Publish/Subscribe System for Large-Scale Science Analytics

Dayal

Bratcher

Eisenhauer

et al. 2014

2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

View full text Add to dashboard Cite

Abstract-As high-end systems move toward exascale sizes, a new model of scientific inquiry being developed is one in which online data analytics run concurrently with the high end simulations producing data outputs. Goals are to gain rapid insights into the ongoing scientific processes, assess their scientific validity, and/or initiate corrective or supplementary actions by launching additional computations when needed. The Flexpath system presented in this paper addresses the fundamental problem of how to structure and efficiently implement the communications between high end simulations and concurrently running online data analytics, the latter comprised of componentized dynamic services and service pipelines.Using a type-based publish/subscribe approach, Flexpath encourages diversity by permitting analytics services to differ in their computational and scaling characteristics and even in their internal execution models. Flexpath uses direct and MxN connections between interacting services to reduce data movements, to allow for runtime connectivity changes to accommodate component arrivals/departures, and to support the multiple underlying communication protocols used for analytics workflows in which simulation outputs are processed by analytics services residing on the same nodes where they are generated, on the same machine, and/or on attached or remote analytics engines. This paper describes the design and implementation of Flexpath, and evaluates it with two widely used scientific applications and their associated data analytics methods.

show abstract

Light-Weight Data Management Solutions for Visualization and Dissemination of Massive Scientific Datasets - Position Paper

Agrawal

2012

2012 SC Companion: High Performance Computing, Networking Storage and Analysis

View full text Add to dashboard Cite

Supporting XML Based High-Level Abstractions on HDF5 Datasets: A Case Study in Automatic Data Virtualization

Cited by 3 publications

References 18 publications

Optimization of PACS Data Persistency Using Indexed Hierarchical Data

Optimization of PACS Data Persistency Using Indexed Hierarchical Data

Flexpath: Type-Based Publish/Subscribe System for Large-Scale Science Analytics

Light-Weight Data Management Solutions for Visualization and Dissemination of Massive Scientific Datasets - Position Paper

Contact Info

Product

Resources

About