Array databases: concepts, standards, implementations

Abstract

Multi-dimensional arrays (also known as raster data or gridded data) play a key role in many, if not all science and engineering domains where they typically represent spatio-temporal sensor, image, simulation output, or statistics “datacubes”. As classic database technology does not support arrays adequately, such data today are maintained mostly in silo solutions, with architectures that tend to erode and not keep up with the increasing requirements on performance and service quality. Array Database systems attempt to close this gap by providing declarative query support for flexible ad-hoc analytics on large n-D arrays, similar to what SQL offers on set-oriented data, XQuery on hierarchical data, and SPARQL and CIPHER on graph data. Today, Petascale Array Database installations exist, employing massive parallelism and distributed processing. Hence, questions arise about technology and standards available, usability, and overall maturity. Several papers have compared models and formalisms, and benchmarks have been undertaken as well, typically comparing two systems against each other. While each of these represent valuable research to the best of our knowledge there is no comprehensive survey combining model, query language, architecture, and practical usability, and performance aspects. The size of this comparison differentiates our study as well with 19 systems compared, four benchmarked to an extent and depth clearly exceeding previous papers in the field; for example, subsetting tests were designed in a way that systems cannot be tuned to specifically these queries. It is hoped that this gives a representative overview to all who want to immerse into the field as well as a clear guidance to those who need to choose the best suited datacube tool for their application. This article presents results of the Research Data Alliance (RDA) Array Database Assessment Working Group (ADA:WG), a subgroup of the Big Data Interest Group. It has elicited the state of the art in Array Databases, technically supported by IEEE GRSS and CODATA Germany, to answer the question: how can data scientists and engineers benefit from Array Database technology? As it turns out, Array Databases can offer significant advantages in terms of flexibility, functionality, extensibility, as well as performance and scalability—in total, the database approach of offering “datacubes” analysis-ready heralds a new level of service quality. Investigation shows that there is a lively ecosystem of technology with increasing uptake, and proven array analytics standards are in place. Consequently, such approaches have to be considered a serious option for datacube services in science, engineering and beyond. Tools, though, vary greatly in functionality and performance as it turns out.

The Open Database of Regional Models of the International Service for the Geoid

“…While partitioning the datasets can be a solution for the management of large raster datasets, the implementation efficiency can vary depending on the partitioning methods. This techniques can vary, and in case of PostGIS Raster, there is the possibility to manage 2D tiles x/y and sets as default a size for the tiling of 100 x 100 pixels (Baumann et al, 2021). It can be considered the use of a multi-dimensional array solution for the management of raster data, especially when the possibility of presenting the data as an n-Dimensional array (accounting for multiple versions of the geoids) is relevant for the community.…”

A Survey on Spatio-temporal Data Analytics Systems

“…The other array databases for processing large-scale spatial raster data are Google Earth Engine [104] and ChronosDB [213,278], in which ChronosDB is the most recent one. We refer to [36,117,279] to learn more about array data databases for processing big spatial raster data.…”

HPC-oriented Canonical Workflows for Machine Learning Applications in Climate and Weather Prediction

“…To address the efficient data management challenge in the DeepRain project, we deployed an arraycentric database. After evaluating several candidates, we opted for the Array Database Management System rasdaman [34] which offers geo-semantic query functionalities for multidimensional arrays, also referred to as datacubes [35]. In parallel to traditional file-based data, rasdaman provides efficient management of large-scale objects, and standardized data modeling [36] which contributes to data harmonization and it allows for flexible access, extraction, analysis, and fusion of massive Spatio-temporal datacubes based on a standardized query language.…”