Hierarchical Storage Support and Management for Large-Scale Multidimensional Array Database Management Systems

Reiner, Bernd; Hahn, Karl; Höfling, Gabriele; Baumann, Peter

doi:10.1007/3-540-46146-9_68

Cited by 24 publications

(13 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Concerning storage optimization, adaptive tiling [12] and compression [10] turn out advantageous. Transparent integration of tertiary storage with emphasis on spatial clustering in tape cabinets has been investigated in [30]. As for processing optimization, several techniques have been shown to speed up response times.…”

Section: Discussionmentioning

confidence: 99%

The OGC web coverage processing service (WCPS) standard

Baumann

2009

Geoinformatica

Self Cite

104

View full text Add to dashboard Cite

Imagery is more and more becoming integral part of geo services. More generally, an increasing variety of sensors is generating massive amounts of data whose quantized nature frequently leads to rasterized data structures. Examples include 1-D time series, 2-D imagery, 3-D image time series and x/y/z spatial cubes, and 4-D x/y/z/t spatio-temporal cubes. The massive proliferation of such raster data through a rapidly growing number of services make open, standardized service interfaces increasingly important. Geo service standardization is undertaken by the Open GeoSpatial Consortium (OGC). The core raster service standard is the Web Coverage Service (WCS) which specifies retrieval based on subsetting, scaling, and reprojection. In 2008, OGC has issued a companion standard which adds flexible, open-ended coverage processing capabilities. This Web Coverage Processing Service (WCPS) specifies a coverage processing language allowing clients to send requests of arbitrary complexity for evaluation by the server. This contribution reports on the WCPS standard by giving an introduction to its coverage model and processing language. Further, design rationales are discussed, as well as background and relation to other OGC standards. 1-D to 4-D use case scenarios illustrate intended use and benefits for different communities. Although the paper focuses on conceptual issues, the WCPS reference implementation, PetaScope, is briefly addressed. The author is co-chair of the coverage-related working groups in OGC.

show abstract

Section: Discussionmentioning

confidence: 99%

The OGC web coverage processing service (WCPS) standard

Baumann

2009

Geoinformatica

Self Cite

104

View full text Add to dashboard Cite

show abstract

“…Since prior systems involved a data ingestion phase, they addressed the problem by reorganizing data in a way that impact of skew was minimized [38,37,34]. Particularly, the techniques included storing the array in irregular chunks of unequal sizes but with the same amount of data, or shuffling the chunks to randomly distribute them across processors.…”

Section: Related Workmentioning

confidence: 99%

Saga

Wang

Nandi

Agrawal

2014

Proceedings of the 26th International Conference on Scientific and Statistical Database Management

View full text Add to dashboard Cite

In recent years, many Array DBMSs, including SciDB and RasDaMan have emerged to meet the needs of data management applications where the natural structures are the arrays. These systems, like their relational counterparts, involve an expensive data ingestion phase. The paradigm of using native storage as a DB and providing database-like support (e.g., the NoDB approach) has recently been shown to be an effective approach for dealing with infrequently queried data, where data ingestion costs cannot be justified, though only in context of relational data.Applications that generate massive arrays, such as the scientific simulations, often store the data in one of a small number of array storage formats, like NetCDF or HDF5. Thus, a natural question is, "can database-like functionality be supported over native array storage?". In this paper, we present algorithms, different partitioning strategies, and an analytical model for supporting structural (grid, sliding, hierarchical, and circular) aggregations over native array storage, and describe implementation of this approach in a system we refer to as Structural AGgregations over Array storage (SAGA). We show how the relative performance of different partitioning strategies changes with varying amount of computation in the aggregation function and different levels of data skew, and our model is effective in choosing the best partitioning strategy. Performance comparison with SciDB shows that despite working on native array storage, the aggregation costs with our system are lower. Finally, we also show that our structural aggregation implementations achieve high parallel efficiency.

show abstract

“…Two-level chunking has been studied before [33,38], but only as a container to place multiple chunks on a single disk block. This approach is a form of IREG-REG, since regular tiles are grouped into irregular chunks.…”

Section: Irregular Chunks (Ireg) Several Schemes Have Also Beenmentioning

confidence: 99%

“…Their conclusions are thus different from ours since they find that arbitrary tiling tuned to a specific workload outperforms regular tiling. Reiner et al [33] studied hierarchical storage support for large-scale multi-dimensional arrays in RasDaMan. Their approach is analogous to the two-level, IREG-REG, chunking strategy.…”

Section: Related Workmentioning

confidence: 99%

ArrayStore

Soroush

Balazinska

Wang

2011

Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data

103

View full text Add to dashboard Cite

We present the design, implementation, and evaluation of ArrayStore, a new storage manager for complex, parallel array processing. ArrayStore builds on prior work in the area of multidimensional data storage, but considers the new problem of supporting a parallel and more varied workload comprising not only range-queries, but also binary operations such as joins and complex user-defined functions.This paper makes two key contributions. First, it examines several existing single-site storage management strategies and array partitioning strategies to identify which combination is best suited for the array-processing workload above. Second, it develops a new and efficient storagemanagement mechanism that enables parallel processing of operations that must access data from adjacent partitions.We evaluate ArrayStore on over 80GB of real data from two scientific domains and real operators used in these domains. We show that ArrayStore outperforms previously proposed storage management strategies in the context of its diverse target workload.

show abstract

Hierarchical Storage Support and Management for Large-Scale Multidimensional Array Database Management Systems

Cited by 24 publications

References 7 publications

The OGC web coverage processing service (WCPS) standard

The OGC web coverage processing service (WCPS) standard

Saga

ArrayStore

Contact Info

Product

Resources

About