Automatic, Efficient and Scalable Provenance Registration for FAIR HPC Workflows

Abstract

Provenance registration is becoming more and more important, as we increase the size and number of experiments performed using computers. In particular, when provenance is recorded in HPC environments, it must be efficient and scalable. In this paper, we propose a provenance registration method for scientific workflows, efficient enough to run in supercomputers (thus, it could run in other environments with more relaxed restrictions, such as distributed ones). It also must be scalable in order to deal with large workflows, that are more typically used in HPC. We also target transparency for the user, shielding them from having to specify how provenance must be recorded. We implement our design using the COMPSs programming model as a Workflow Management System (WfMS) and use RO-Crate as a well-established specification to record and publish provenance. Experiments are provided, demonstrating the run time efficiency and scalability of our solution.

Julia as a unifying end-to-end workflow language on the Frontier exascale system

“…This complexity is primarily driven by two factors: (i) the evolving application requirements for experimental, observational, and computational science and (ii) the extreme heterogeneity of our computing and data generation and processing systems [11,16,43]. Consequently, the community vision for a research and development roadmap [14,15] has identified key challenges posed by integrating HPC, AI, and FAIR (findable, accessible, interoperable, and reusable) [18,41] workflows at exascale.…”