rasdaman

Baumann, Peter; Mišev, Dimitar; Merticariu, Vlad; Huu, Bang Pham; Bell, Brennan

doi:10.1145/3274895.3274988

Cited by 11 publications

(1 citation statement)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, datacubes take advantage of the regular structure of 'raster' or 'gridded' ESD to organize, combine, and scale-up essentially by stacking arrays and then partitioning them on cloud. This is exemplified by database-like approaches such as rasdaman and the arbitrary-size-array cloud infrastructure approach of Pangeo [4,5]. Both approaches provide scalable ways to analyze ESD data, but both require (diverse) ESD to be first interpolated (gridded), arguably demanding all users work with L3-like data.…”

Section: Introductionmentioning

confidence: 99%

STARE-based Integrative Analysis of Diverse Data Using Dask Parallel Programming Demo Paper

Rilee¹,

Griessbaum

Kuo³

et al. 2020

Proceedings of the 28th International Conference on Advances in Geographic Information Systems

View full text Add to dashboard Cite

Scaling up volume and variety in Big Earth Science Data is particularly difficult when combining low-level, ungridded data, such as swath observations obtained with, for example, Moderate Resolution Imaging Spectroradiometers (MODIS). A unified way to index and combine data with different geo-spatiotemporal layouts and incomparable native array formatting is required for scalable integrative analyses based on data at its full instrument resolution, that is, without extra interpolation (or extrapolation) onto a common grid. The SpatioTemporal Adaptive Resolution Encoding (STARE) uses the Hierarchical Triangular Mesh (HTM) and the Hierarchical Calendrical Partitioning (HCP), recursive partitionings of solid angle and time into tree data structures, to encode spatiotemporal neighborhoods as sets of integers. Regions sharing common paths through the STARE tree hierarchy have similar index values, which can then serve as keys in algorithms and data structures supporting scalable integrative analyses. Thus, STARE co-aligns data in both physical (spatiotemporal) and cyber (memory) spaces, providing a means for marshalling computing resources and conducting analysis with minimum data movement, addressing volume scalability while simultaneously unifying diverse data for variety scaling. In this paper, we demonstrate how easy it is to use the Python STARE API (PySTARE) and the parallel programming platform Dask to integrate MODIS and Geostationary Operational Environmental Satellite (GOES) data, datasets with very different geo-spatiotemporal characteristics. CCS CONCEPTS •Computing methodologies~Parallel computing methodologies•Applied computing~Physical sciences and engineering~Earth and atmospheric sciences•Information systems~Data management systems~Information integration

show abstract

Section: Introductionmentioning

confidence: 99%

STARE-based Integrative Analysis of Diverse Data Using Dask Parallel Programming Demo Paper

Rilee¹,

Griessbaum

Kuo³

et al. 2020

Proceedings of the 28th International Conference on Advances in Geographic Information Systems

View full text Add to dashboard Cite

show abstract

Querying Large Scientific Data Sets with Adaptable IO System ADIOS

Gu¹,

Klasky²,

Podhorszki³

et al. 2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. When working with a large dataset, a relatively small fraction of data records are of interest in each analysis operation. For example, while examining a billion-particle dataset from an accelerator model, the scientists might focus on a few thousand fastest particles, or on the particle farthest from the beam center. In general, this type of selective data access is challenging because the selected data records could be anywhere in the dataset and require a significant amount of time to locate and retrieve. In this paper, we report our experience of addressing this data access challenge with the Adaptable IO System ADIOS. More specifically, we design a query interface for ADIOS to allow arbitrary combinations of range conditions on known variables, implement a number of different mechanisms for resolving these selection conditions, and devise strategies to reduce the time needed to retrieve the scattered data records. In many cases, the query mechanism can retrieve the selected data records orders of magnitude faster than the brute-force approach.Our work relies heavily on the in situ data processing feature of ADIOS to allow user functions to be executed in the data transport pipeline. This feature allows us to build indexes for efficient query processing, and to perform other intricate analyses while the data is in memory.

show abstract

Cloud-based storage and computing for remote sensing big data: a technical review

Fan

et al. 2022

International Journal of Digital Earth

View full text Add to dashboard Cite

rasdaman

Cited by 11 publications

References 4 publications

STARE-based Integrative Analysis of Diverse Data Using Dask Parallel Programming Demo Paper

STARE-based Integrative Analysis of Diverse Data Using Dask Parallel Programming Demo Paper

Querying Large Scientific Data Sets with Adaptable IO System ADIOS

Cloud-based storage and computing for remote sensing big data: a technical review

Contact Info

Product

Resources

About