“…Probabilistic deep learning has emerged as a key component in the application of machine learning in the geosciences. These efforts include importing general computer/data science tools into the geosciences (e.g., generative adversarial networks), as well as the development of new tools for specific geoscience use‐cases (e.g., Crommelin & Edeling, 2021; Dorling et al., 2003; Foster et al., 2021; Gagne et al., 2020; Guillaumin & Zanna, 2021; Leinonen et al., 2020; Scher & Messori, 2021). In this paper, we do both.…”
The earth system is exceedingly complex and often chaotic in nature, making prediction incredibly challenging: we cannot expect to make perfect predictions all of the time. Instead, we look for specific states of the system that lead to more predictable behavior than others, often termed “forecasts of opportunity.” When these opportunities are not present, scientists need prediction systems that are capable of saying “I don't know.” We introduce a novel loss function, termed the “NotWrong loss,” that allows neural networks to identify forecasts of opportunity for classification problems. The NotWrong loss introduces an abstention class that allows the network to identify the more confident samples and abstain (say “I don't know”) on the less confident samples. The abstention loss is designed to abstain on a user‐defined fraction of the samples via a standard adaptive controller. Unlike many machine learning methods used to reject samples post‐training, the NotWrong loss is applied during training to preferentially learn from the more confident samples. We show that the NotWrong loss outperforms other existing loss functions for multiple climate use cases. The implementation of the proposed loss function is straightforward in most network architectures designed for classification as it only requires the addition of an abstention class to the output layer and modification of the loss function.
“…Probabilistic deep learning has emerged as a key component in the application of machine learning in the geosciences. These efforts include importing general computer/data science tools into the geosciences (e.g., generative adversarial networks), as well as the development of new tools for specific geoscience use‐cases (e.g., Crommelin & Edeling, 2021; Dorling et al., 2003; Foster et al., 2021; Gagne et al., 2020; Guillaumin & Zanna, 2021; Leinonen et al., 2020; Scher & Messori, 2021). In this paper, we do both.…”
The earth system is exceedingly complex and often chaotic in nature, making prediction incredibly challenging: we cannot expect to make perfect predictions all of the time. Instead, we look for specific states of the system that lead to more predictable behavior than others, often termed “forecasts of opportunity.” When these opportunities are not present, scientists need prediction systems that are capable of saying “I don't know.” We introduce a novel loss function, termed the “NotWrong loss,” that allows neural networks to identify forecasts of opportunity for classification problems. The NotWrong loss introduces an abstention class that allows the network to identify the more confident samples and abstain (say “I don't know”) on the less confident samples. The abstention loss is designed to abstain on a user‐defined fraction of the samples via a standard adaptive controller. Unlike many machine learning methods used to reject samples post‐training, the NotWrong loss is applied during training to preferentially learn from the more confident samples. We show that the NotWrong loss outperforms other existing loss functions for multiple climate use cases. The implementation of the proposed loss function is straightforward in most network architectures designed for classification as it only requires the addition of an abstention class to the output layer and modification of the loss function.
“…To address the more fundamental model-form uncertainty, we are currently investigating the use of data-driven stochastic surrogates to replace traditional deterministic parametrizations; see e.g. [ 55 , 56 ] for recent results. Figure 7 A Tube Map showing the VECMA components used in the climate application.…”
We present the VECMA toolkit (VECMAtk), a flexible software environment for single and multiscale simulations that introduces directly applicable and reusable procedures for verification, validation (V&V), sensitivity analysis (SA) and uncertainty quantication (UQ). It enables users to verify key aspects of their applications, systematically compare and validate the simulation outputs against observational or benchmark data, and run simulations conveniently on any platform from the desktop to current multi-petascale computers. In this sequel to our paper on VECMAtk which we presented last year [
1
] we focus on a range of functional and performance improvements that we have introduced, cover newly introduced components, and applications examples from seven different domains such as conflict modelling and environmental sciences. We also present several implemented patterns for UQ/SA and V&V, and guide the reader through one example concerning COVID-19 modelling in detail.
This article is part of the theme issue ‘Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification
in silico
’.
“…A third approach is a Linear Inverse Modeling framework (Newman et al, 2003;Martinez-Villalobos et al, 2017), where the predictive modes are represented as covariance functions in a reduced space (e.g., functions of PCAs). We can also model the system with reduced complexity and represent higher complexity processes as AI-driven stochastic processes (Chattopadhyay et al, 2020;Crommelin and Edeling, 2020;Alcala and Timofeyev, 2020;Leinonen et al, 2020). To characterize noise relevant for predicting high-frequency signals, convection-resolving simulations such as the DYAMOND ensemble (Stevens et al, 2019) provide comprehensive data coverage to characterize variability in small-scale processes (Christensen, 2020).…”
Section: (A) the Stochastic Surrogate Modelsmentioning
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
