Data Compression-Aware Performance Analysis of Dask and Spark for Earth Observation Data Processing

Abstract

High-performance computing is a good choice for handling Big Earth Observation data, allowing the processing of the data in a distributed and performance-efficient way using in-memory computing frameworks. The data compression technique reduces the amount of storage and network transfer time and improves processing performance. The article aims to investigate the effectiveness of widely used distributed data processing frameworks in conjunction with lossless data compression techniques, to find the optimal compression method and processing framework for specific earth observation workflows. Normalized Difference Vegetation Index has been evaluated for the territory of Armenia, obtaining data from the Sentinel satellite and considering the supported compression methods to compare the performance of in-memory Dask and Spark frameworks. Experiments show that the Zstandard compression method and the Dask framework are the best choices for such workflows.

Enhancing Earth Observation Data Processing through Optimized Multi-Modular Service

“…This examination compares the effects of the Dask and Spark environments on the speed of data processing. Study shows [13] that Dask and Spark both offer comparable data processing performance. However, combining the Dask environment with the Zstandard compression technique yields the best performance results.…”