USFD: a unified storage framework for SOAR HPC scientific workflows

Abstract

Emerging scientific workflows in high performance computing (HPC) focus more on analysis rather than on simulation. Simulation output is so dense with information that copious amounts of analysis must be performed on a single output to understand the results of that simulation. We identify this repetitive analysis as a new application type, simulate once analyse repeatedly (SOAR) computing. Current scientific HPC, when extended to SOAR computing, results in excessive data migration between compute and storage resources. For a workflow bound by file I/O, a large data migration overhead is unacceptable. We propose a framework that uses a data-intensive storage cluster coupled with an interoperability layer, called unified storage framework designed (USFD).USFD is a better support SOAR HPC scientific workloads through enhanced file I/O support and co-located storage and analysis. In this work, we analyse the performance of USFD and other traditional HPC approaches for SOAR scientific workloads. Our results show that SOAR workflows using USFD complete 7.5 times faster over other approaches with quantum chromodynamic workflows and 4 times faster with FLASH workflows.