Database system support of simulation data

Abstract

Supported by increasingly efficient HPC infrastructure , numerical simulations are rapidly expanding to fields such as oil and gas, medicine and meteorology. As simulations become more precise and cover longer periods of time, they may produce files with terabytes of data that need to be efficiently analyzed. In this paper, we investigate techniques for managing such data using an array DBMS. We take advantage of multidimensional arrays that nicely models the dimensions and variables used in numerical simulations. However, a naive approach to map simulation data files may lead to sparse arrays, impacting query response time, in particular, when the simulation uses irregular meshes to model its physical domain. We propose efficient techniques to map coordinate values in numerical simulations to evenly distributed cells in array chunks with the use of equi-depth histograms and space-filling curves. We implemented our techniques in SciDB and, through experiments over real-world data, compared them with two other approaches: row-store and column-store DBMS. The results indicate that multidimensional arrays and column-stores are much faster than a traditional row-store system for queries over a larger amount of simulation data. They also help identifying the scenarios where array DBMSs are most efficient, and those where they are outperformed by column-stores.

Managing Sparse Spatio-Temporal Data in SAVIME: an Evaluation of the Ph-tree Index

“…Since SciDB splits the array into chunks, which are equally sized subarrays, it becomes necessary to define a balanced partitioning scheme when dealing with irregularly distributed data. For sparse arrays, this is seldom a trivial task, since a poor partitioning scheme may impact query response time, greatly affecting database performance [Lustosa et al 2016]. Rasdaman [Baumann et al 1998] was a pioneer array DBMS that estabilished arrays as first-class database structures.…”

“…A few Array data processing systems have appeared recently offering a data model and query algebra adequate for representing and processing non-Relational data [Baumann et al 1998, Stonebraker et al 2011, Lustosa et al 2016, Zalipynis 2018]. In particular, SciDB was conceived having large scale scientific data management as target applications [Brown 2010], such as data produced by the Sloan Digital Sky Survey.…”

“…Managing sparse arrays is a challenge for array data processing systems [Lustosa et al 2016]. This is due to two main factors: (i) if a dense data structure is used, there will be many empty cells which will hamper the query access time while allocating huge non-used memory and disk space; (ii) conversely, when a sparse allocation is adopted, the coordinate system indexes are allocated as cell attributes, which jeopardizes the performance of index-based queries.…”

SAVIME: An Array DBMS for Simulation Analysis and ML Models Prediction

“…Our previous study on the usage of DBMSs to manage numerical simulation data [Lustosa et al 2016] highlighted many difficulties regarding scientific array data representation. We concluded that the idiosyncrasies of this kind of data are not well represented by current array data models.…”

Towards In-transit Analysis on Supercomputing Environments

“…For each time step previously defined, the mesh is updated with new values for its vertices and edges. A native representation for this kind of data are multidimensional matrices [Lustosa et al 2016].…”