HDF5-FastQuery: Accelerating Complex Queries on HDF Datasets using Fast Bitmap Indices

Abstract

Large scale scientific data is often stored in scientific data formats such as FITS, netCDF and HDF. These storage formats are of particular interest to the scientific user community since they provide multi-dimensional storage and retrieval. However, one of the drawbacks of these storage formats is that they do not support semantic indexing which is important for interactive data analysis where scientists look for features of interests such as "Find all supernova explosions where energy > 10 5 and temperature > 10 6 ".In this paper we present a novel approach called HDF5-FastQuery to accelerate the data access of large HDF5 files by introducing multi-dimensional semantic indexing. Our implementation leverages an efficient indexing technology called bitmap indexing that has been widely used in the database community. Bitmap indices are especially well suited for interactive exploration of large-scale readonly data. Storing the bitmap indices into the HDF5 file has the following advantages: a) Significant performance speedup of accessing subsets of multi-dimensional data and b) portability of the indices across multiple computer platforms. We will present an API that simplifies the execution of queries on HDF5 files for general scientific applications and data analysis. The design is flexible enough to accommodate the use of arbitrary indexing technology for semantic range queries. We will also provide a detailed performance analysis of HDF5-FastQuery for both synthetic and scientific data. The results demonstrate that our proposed approach for multi-dimensional queries is up to a factor of 2 faster than HDF5.

FastQuery: A Parallel Indexing System for Scientific Data

“…This behavior has been observed and explained in the previously study in the HDF5-FastQuery paper [9]. The reason is that as the selectivity increases, more bitmaps need to be read from file and processed by FastBit.…”

“…This would make it necessary for us to work with subarrays. Earlier work on indexing scientific data have only considered indexing whole arrays [9]; we believe that we are the first ones to consider indexing subarrays. We will describe how to address the mapping of rows and tables in the next section.…”

“…As shown by our experimental evaluations in Section 4.2, without the support of array specification (i.e. such as using HDF5-FastQuery [9]), the post-processing overhead could easily dominate the overall query response time and limits the performance benefit from using an indexing/query system. The approach of FastQuery is to allow users explicitly specify data with an array description through the FastQuery API.…”

Progress on H5Part: a portable high performance parallel data interface for electromagnetics simulations

“…We are currently working on integrating the FastBit bitmap indexing technology [4] for accelerating queries of data stored in H5Part. An example of such a query is (p x > 1.34) AND (potential < 2e7).…”

Automatic beam path analysis of laser wakefield particle acceleration data

“…Being able to efficiently access the relevant data subsets is crucial to enable good computational performance. We augment the data with FastBit bitmap indices and use HDF5-FastQuery [52] -a data access API based on HDF5 [53], H5Part [54,55] and FastBit [9]-to enable efficient access to the data. The same data interface was used in earlier work to enable high-performance visualization of LWFA simulation data [7].…”