Querying Large Scientific Data Sets with Adaptable IO System ADIOS

Abstract

Abstract. When working with a large dataset, a relatively small fraction of data records are of interest in each analysis operation. For example, while examining a billion-particle dataset from an accelerator model, the scientists might focus on a few thousand fastest particles, or on the particle farthest from the beam center. In general, this type of selective data access is challenging because the selected data records could be anywhere in the dataset and require a significant amount of time to locate and retrieve. In this paper, we report our experience of addressing this data access challenge with the Adaptable IO System ADIOS. More specifically, we design a query interface for ADIOS to allow arbitrary combinations of range conditions on known variables, implement a number of different mechanisms for resolving these selection conditions, and devise strategies to reduce the time needed to retrieve the scattered data records. In many cases, the query mechanism can retrieve the selected data records orders of magnitude faster than the brute-force approach.Our work relies heavily on the in situ data processing feature of ADIOS to allow user functions to be executed in the data transport pipeline. This feature allows us to build indexes for efficient query processing, and to perform other intricate analyses while the data is in memory.

UniIndex: An index and query middleware for parallel file systems

“…Second, the decoupled approach alleviates the metadata bottleneck of metadata servers of the underlying PFS since query operations can be handled by the external query manager. Finally, as previous studies have demonstrated, scientific data are rarely modified once they are generated, and data consistency between external KV pairs and data files can be easily maintained.…”

“…It may complicate the consistency issues by managing UDM and indexes as external KV pairs. However, as previous studies have demonstrated, scientific data are rarely modified once they are generated, we apply a relaxed consistency model in our current implementation. UniIndex provides encapsulated APIs (eg, setAttribute and buildIndex) and command line utilities to update external metadata objects.…”

“…FastQuery applies bitmap indexing techniques and provides parallel query frameworks to accelerate data selection on HDF5 and NetCDF data formats. Previous works by Wu et al designed a query interface for ADIOS library to allow arbitrary combinations of range conditions on known variables. However, these works store the indexes in a separate index file, which may lead to potential write contention and extra index load costs when processing query operations .…”

“…FastQuery 15 applies bitmap indexing techniques and provides parallel query frameworks to accelerate data selection on HDF5 and NetCDF data formats. Previous works by Wu et al 16,17 designed a query interface for ADIOS 19 library to allow arbitrary combinations of range conditions on known variables.…”

“…There are also some research efforts that have been made to add index and query interface to high-level I/O libraries. 15,16,17 However, these works write indexes to a separate index file, which may lead to the potential write contention and I/O overhead. 18 Driven by the problems mentioned above, we propose two mechanisms to provide efficient data locating services at the granularity of both a file and a record.…”