HPC-oriented Canonical Workflows for Machine Learning Applications in Climate and Weather Prediction

Mozaffari, Amirpasha; Langguth, Michael; Gong, Bing; Ahring, Jessica; Campos, Adrian Rojas; Nieters, Pascal; Escobar, Otoniel José Campos; Wittenbrink, Martin; Baumann, Peter; Schultz, Martin

doi:10.1162/dint_a_00131

Cited by 6 publications

(2 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to address the challenges with respect to data mentioned above, current efforts at DKRZ (German Climate Computing Center) are aimed at a complete restructuring of the way research is performed in simulation-based climate research (Anders et al 2022, Mozaffari et al 2022, Weigel et al 2020. DKRZ is perfectly suited for this endeavor, because researchers have the resources and services available to conduct the entire suite of their data-intensive workflows -ranging from planning and setting up of model simulations, analyzing the model output, reusing existing large-volume datasets to data publication and long-term archival.…”

Section: Cultural Changementioning

confidence: 99%

Facing the Challenges in simulation-based Earth System Sciences and the Role of FAIR Digital Objects

Anders¹,

Gehlen²,

Bergemann³

et al. 2022

RIO

View full text Add to dashboard Cite

Motivation Results of simulations with climate models form the most important basis for research and statements about possible changes in the future global, regional and local climate. These output volumes are increasing at an exponential rate (Balaji et al. 2018, Stevens et al. 2019). Efficiently handling these amounts of data is a challenge for researchers, mainly because the development of novel data and workflow handling approaches have not proceeded at the same rate as data volume has been increasing. This problem will only become more pronounced with the ever increasing performance of High Performance Computing (HPC) - systems used to perform weather and climate simulations (Lawrence et al. 2018). For example, in the framework of the European Commission's Destination Earth program the Digital Twins (Bauer et al. 2021) are expected to produce hundreds of terabytes of model output data every day at the EuroHPC computing sites. The described data challenge can be dissected into several aspects, two of which we will focus on in this contribution. Available data in the Earth System Sciences (ESS) are increasingly made openly accessible by various institutions, such as universities, research centres and government agencies, in addition to subject-specific repositories. Further, the exploitability of weather and climate simulation output beyond the expert community by humans and automated agents (as described by the FAIR data principles (F-Findable, A-Accessable, I-Interoperable, R-Reusable), Wilkinson et al. 2016) is currently very limited if not impossible due to disorganized metadata or incomplete provenance information. Additionally, developments regarding globally available and FAIR workflows in the spirit of the FAIR Digital Object (FDO) framework (Schultes and Wittenburg 2019, Schwardmann 2020) are just at the beginning. Cultural Change In order to address the challenges with respect to data mentioned above, current efforts at DKRZ (German Climate Computing Center) are aimed at a complete restructuring of the way research is performed in simulation-based climate research (Anders et al. 2022, Mozaffari et al. 2022, Weigel et al. 2020). DKRZ is perfectly suited for this endeavor, because researchers have the resources and services available to conduct the entire suite of their data-intensive workflows - ranging from planning and setting up of model simulations, analyzing the model output, reusing existing large-volume datasets to data publication and long-term archival. At the moment, DKRZ-users do not have the possibility to orchestrate their workflows via a central service, but rather use a plethora of different tools to piece them together. Framework Environment Freva The central element of the new workflow environment at DKRZ shall be represented by the Freva (Free Evaluation System Framework) software infrastructure, which offers standardized data and tool solutions in ESS and is optimized for use on high-performance computer systems (Kadow et al. 2021). Freva is designed to be very well suited to the use of the FDO framework. The crucial aspects here are: the standardisation of data objects as input for analysis and processing, the already implemented remote access to data via a Persisitent Identifier (PID), the currently still system-internal capture of analysis provenance and the possibility of sharing results but also workflows by research groups up to large communities. the standardisation of data objects as input for analysis and processing, the already implemented remote access to data via a Persisitent Identifier (PID), the currently still system-internal capture of analysis provenance and the possibility of sharing results but also workflows by research groups up to large communities. It is planned to extend the functionality of Freva so that the system automatically determines the data required for a specific analysis from a researcher’s research question (provided to the system via some interface), enquires available databases (local disk or tape, cloud or federated resources) for that data and retrieves the data if possible. If data are not available (yet), Freva shall be able to automatically configure, set up and submit model simulations to the HPC-System, so that the required data is created and becomes available (cf. Fig. 1). These data will in turn be ingested into Freva’s data catalog for reuse. Next, Freva shall orchestrate and document the analysis performed. Results will be provided either as numerical fields, images or animations depending on the researcher’s need. As a final step, the applied workflow and/or underlying data are published in accordance with the FAIR data guiding principles. FDOs - towards a global integrated Data Space To make the process sketched out above a reality, application of the FDO concept is essential (Schwardmann 2020, Schultes and Wittenburg 2019). There is a long tradition in the ESS community of global dissemination and reuse of large-volume climate data sets. Community standards like those developed and applied in the framework of internationally coordinated model intercomparison studies (CMIP) allow for low-barrier reuse of data (Balaji et al. 2018). Globally resolvable PIDs are provided on a regular basis. Current community ESS standards and workflows are already close to being compatible with implementing FDOs, however, now we also have to work on open points in the FDO concept, which are: the clear definition of community-specific FDO requirements including PID Kernel Types specifications, the operation of data type registries and the technical implementation requirements for global access to FDOs. the clear definition of community-specific FDO requirements including PID Kernel Types specifications, the operation of data type registries and the technical implementation requirements for global access to FDOs. With these in place and implemented in Freva following standardized implementation recommendations, automated data queries across spatially distributed or different types of local databases become possible. We introduce the concept of implementations in Freva and also use it to highlight the challenges we face. Using an example, we show the vision of the work of a scientist in earth system science.

show abstract

Section: Cultural Changementioning

confidence: 99%

Facing the Challenges in simulation-based Earth System Sciences and the Role of FAIR Digital Objects

Anders¹,

Gehlen²,

Bergemann³

et al. 2022

RIO

View full text Add to dashboard Cite

show abstract

“…At the beginning, all data mining samples are in the root node, and using a recursive approach, the samples are divided by specified attributes, and a corresponding number of branches are developed based on the attribute value, which must be discrete, or if the attribute value is continuous, it must first be discretized by processing this attribute value to make it discrete [11][12]. In the second stage of decision tree pruning, it is to cut out the isolated points and noise generated in the training data, so that the constructed decision tree can be compared one by one with each attribute value of the sample data, so as to achieve the purpose of classifying the unknown sample data and mining it [13].…”

Section: Decision Treesmentioning

confidence: 99%

Predicting the Employment Destinations of University Students Based on Machine Learning Algorithms

2022

View full text Add to dashboard Cite

The research on the classification prediction of students' employment destinations in higher education not only opens up new application areas for classification algorithms, but also represents a new attempt to introduce machine learning algorithms into the analysis of employment guidance and the development of teaching systems. The purpose of this paper is to study the employment destination prediction of college students based on machine learning algorithms. The factors influencing the employability of university students are analysed and the XGBoost model in decision trees is explored. A graduate employment prediction algorithm based on HMIGW feature selection and XGBoost algorithm is proposed to predict the employment situation of the class of 2022 and the type of employment, and the experimental results show that the algorithm is able to obtain relatively accurate conclusions on graduate employment prediction.

show abstract

Artificial intelligence for climate prediction of extremes: State of the art, challenges, and future perspectives

Materia,

García,

van Straaten

et al. 2024

WIREs Climate Change

View full text Add to dashboard Cite

Extreme events such as heat waves and cold spells, droughts, heavy rain, and storms are particularly challenging to predict accurately due to their rarity and chaotic nature, and because of model limitations. However, recent studies have shown that there might be systemic predictability that is not being leveraged, whose exploitation could meet the need for reliable predictions of aggregated extreme weather measures on timescales from weeks to decades ahead. Recently, numerous studies have been devoted to the use of artificial intelligence (AI) to study predictability and make climate predictions. AI techniques have shown great potential to improve the prediction of extreme events and uncover their links to large‐scale and local drivers. Machine and deep learning have been explored to enhance prediction, while causal discovery and explainable AI have been tested to improve our understanding of the processes underlying predictability. Hybrid predictions combining AI, which can reveal unknown spatiotemporal connections from data, with climate models that provide the theoretical foundation and interpretability of the physical world, have shown that improving prediction skills of extremes on climate‐relevant timescales is possible. However, numerous challenges persist in various aspects, including data curation, model uncertainty, generalizability, reproducibility of methods, and workflows. This review aims at overviewing achievements and challenges in the use of AI techniques to improve the prediction of extremes at the subseasonal to decadal timescale. A few best practices are identified to increase trust in these novel techniques, and future perspectives are envisaged for further scientific development.This article is categorized under: Climate Models and Modeling > Knowledge Generation with Models The Social Status of Climate Change Knowledge > Climate Science and Decision Making

show abstract

HPC-oriented Canonical Workflows for Machine Learning Applications in Climate and Weather Prediction

Cited by 6 publications

References 30 publications

Facing the Challenges in simulation-based Earth System Sciences and the Role of FAIR Digital Objects

Facing the Challenges in simulation-based Earth System Sciences and the Role of FAIR Digital Objects

Predicting the Employment Destinations of University Students Based on Machine Learning Algorithms

Artificial intelligence for climate prediction of extremes: State of the art, challenges, and future perspectives

Contact Info

Product

Resources

About