Optimal Strategies for Storing Earth Science Datasets in the Commercial Cloud

Nguyen, Dieu My T; Cortes, Johana Chazaro; Dunn, Marina; Shiklomanov, Alexey

doi:10.1002/essoar.10511054.1

Cited by 1 publication

(1 citation statement)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The majority of contemporary storage formats for multidimensional arrays, such as HDF5, NetCDF, TIFF, and Zarr, provide the ability to manage the storage and compression of data; however, these options can yield notable performance consequences. The solution in Reference 34 explored different strategies for chunking the data into units of different sizes across the dataset's temporal and spatial dimensions. They explored the performance of various strategies for chunking a multi‐dimensional Earth science dataset by comparing the processing time and memory usage for common data access operations using a single processor to resemble simplified use cases.…”

Section: Related Workmentioning

confidence: 99%

A novel approach for large‐scale environmental data partitioning on cloud and on‐premises storage for compute continuum applications

Mellone,

De Vita,

Sánchez‐Gallegos

et al. 2023

Concurrency and Computation

View full text Add to dashboard Cite

SummaryCloud‐based services have proved useful in several research fields, such as engineering, health science, and astrophysics, to mention a few examples. The computational environmental science community developed a strong need for cloud facilities to store, process, and manage data from observations and numerical models for simulations and forecasts. Weather forecast models and global sensor networks deal with multidimensional geo‐referenced data∖sets. However, environmental data consumer applications usually require a relatively small amount of multidimensional input data slice to analyze a specific area or time interval. Hence, reducing data dimension for information retrieval is mandatory. This paper presents a twofold solution: a technique to load and retrieve the sliced multidimensional data set on different cloud services such as Amazon Web Service (AWS), Google Cloud Platform, and Microsoft Azure. The experimental results performed on these cloud services highlight that the proposed method can significantly speed up the process of loading and retrieving the data slices compared to working with the entire data set in bulk or OPeNDAP server.

show abstract

Section: Related Workmentioning

confidence: 99%