ArrayBridge: Interweaving Declarative Array Processing in SciDB with Imperative HDF5-Based Programs

Abstract

Scientists are increasingly turning to datacenter-scale computers to produce and analyze massive arrays. Despite decades of database research that extols the virtues of declarative query processing, scientists still write, debug and parallelize imperative HPC kernels even for the most mundane queries. This impedance mismatch has been partly attributed to the cumbersome data loading process; in response, the database community has proposed in situ mechanisms to access data in scientific file formats. Scientists, however, desire more than a passive access method that reads arrays from files. This paper describes ArrayBridge, a bi-directional array view mechanism for scientific file formats, that aims to make declarative array manipulations interoperable with imperative file-centric analyses. Our prototype implementation of ArrayBridge uses HDF5 as the underlying array storage library and seamlessly integrates into the SciDB open-source array database system. In addition to fast querying over external array objects, ArrayBridge produces arrays in the HDF5 file format just as easily as it can read from it. Ar-rayBridge also supports time travel queries from imperative kernels through the unmodified HDF5 API, and automatically deduplicates between array versions for space efficiency. Our extensive performance evaluation in NERSC, a large-scale scientific computing facility, shows that Array-Bridge exhibits statistically indistinguishable performance and I/O scalability to the native SciDB storage engine.

A Novel Approach to Translate Structural Aggregation Queries to MapReduce Code

“…For instance, SciDB [13] supports both an SQL-like query language AQL and a functional language AFL (Array Functional Language), RasDaMan uses RasQL [8], and MonetDB initially used both RAM [39] and SciQL [15]. More recent systems are Chronosdb [40] and ArrayBridge [41].…”

SAVIME: An Array DBMS for Simulation Analysis and ML Models Prediction

“…Due to the fact that ingesting data into these systems is not an easy task, other arrays systems, such as ArrayBridge [Xing et al 2018] and ChronosDB [Zalipynis 2018] have been developed. ArrayBridge works over SciDB, and gives it the ability to work directly with HDF5 files.…”

SAVIME: A Database Management System for Simulation Data Analysis and Visualization

“…Many array DBMSs have been proposed, the most prominent ones are RasDaMan [Baumann et al 1997] and, more rencently, SciDB [Cudre-Mauroux et al 2009]. Due to the fact that ingesting data into these systems is not an easy task, other arrays systems, such as ArrayBridge [Xing et al 2017] and ChronoDB [Zalipynis 2018] have been developed. ArrayBridge works over SciDB, and gives it the ability to work directly with HDF5 files.…”