Towards an Open (Data) Science Analytics-Hub for Reproducible Multi-Model Climate Analysis at Scale

Abstract

Open Science is key to future scientific research and promotes a deep transformation in the whole scientific research process encouraging the adoption of transparent and collaborative scientific approaches aimed at knowledge sharing. Open Science is increasingly gaining attention in the current and future research agenda worldwide. To effectively address Open Science goals, besides Open Access to results and data, it is also paramount to provide tools or environments to support the whole research process, in particular the design, execution and sharing of transparent and reproducible experiments, including data provenance (or lineage) tracking. This work introduces the Climate Analytics-Hub, a new component on top of the Earth System Grid Federation (ESGF), which joins big data approaches and parallel computing paradigms to provide an Open Science environment for reproducible multi-model climate change data analytics experiments at scale. An operational implementation has been set up at the SuperComputing Centre of the Euro-Mediterranean Center on Climate Change, with the main goal of becoming a reference Open Science hub in the climate community regarding the multi-model analysis based on the Coupled Model Intercomparison Project (CMIP).

A Graph Data Model-based Micro-Provenance Approach for Multi-level Provenance Exploration in End-to-End Climate Workflows

“…The service has been deployed at CMCC and tested on several use cases in the climate domain. Future work on this topic concerns (i) the knowledge extraction part, by focusing on graph mining algorithms, including learning aspects, (ii) the development of a microservice (container-based) version of yProv for cloud/HPCenabled environments (i.e., ENES Data Space [24][28] [32]), as well as (iii) the service registration into the EOSC Marketplace for a wider adoption both within the climate community and across scientific communities. Finally, (iv) the extension of the graph data model is also envisaged to include system-level provenance resources (i.e., containers) that will be essential to better address provenance (in terms of more complete documentation), computational reproducibility (in terms of virtualized resources running in cloud environments) and to spot issues related to the surrounding software ecosystem (rather than the specific task or application) [27].…”

A Data Space for Climate Science in the European Open Science Cloud

“…Several efforts have been undertaken over the years; for example in the context of the IS-ENES projects a set of compute services have been established by key European climate research centers. Initial ideas for these computing capabilities are represented by the analytics-hub concept 7 and the ENES Climate Analytics Service (ECAS) set up in the context of the EOSC initiative by CMCC and DRKZ. 8 ECAS represented an early solution for providing data analysis capabilities close to the ESGF data pools already deployed in the data center, while the analytics-hub focused on providing analytics service on variable-centric data collections.…”

PyOphidia: A Python library for High Performance Data Analytics at scale

“…The Python module has been used, jointly with the Ophidia serverside components, in a wide set of applications, mainly related to climate and geosciences [28][29][30] but also linked with biodiversity and smart cities [31,32]. Other than being used by scientists on a daily basis, the module has also been exploited in several applications developed in the context of international and European projects.…”