Performance comparison of Dask and Apache Spark on HPC systems for neuroimaging

Abstract

SummaryThe general increase in data size and data sharing motivates the adoption of Big Data strategies in several scientific disciplines. However, while several options are available, no particular guidelines exist for selecting a Big Data engine. In this paper, we compare the runtime performance of two popular Big Data engines with Python APIs, Apache Spark, and Dask, in processing neuroimaging pipelines. Our experiments use three synthetic neuroimaging applications to process the 606 GB BigBrain image and an actual pipeline to process data from thousands of anatomical images. We benchmark these applications on a dedicated HPC cluster running the Lustre file system while using varying combinations of the number of nodes, file size, and task duration. Our results show that although there are slight differences between Dask and Spark, the performance of the engines is comparable for data‐intensive applications. However, Spark requires more memory than Dask, which can lead to slower runtime depending on configuration and infrastructure. In general, the limiting factor was the data transfer time. While both engines are suitable for neuroimaging, more efforts need to be put to reduce the data transfer time and the memory footprint of applications.

Adaptive Cross-platform Scheduling Framework for NWP in Hybrid Clouds

“…However, adapting MPI-dependent models to Spark involves significant changes to exploit cloud computing's high-performance capabilities effectively [49][50][51]. Efforts to integrate Spark with existing HPC architectures or modify it for enhanced performance are ongoing, with research focusing on extending Spark's utility for complex, high-throughput computing tasks typically handled by MPI [52]. This transition highlights the necessity of adapting high-performance computing paradigms to fit hybrid cloud environments, ensuring efficient data handling and computation.…”