Controlled Abstention Neural Networks for Identifying Skillful Predictions for Classification Problems

Abstract

The earth system is exceedingly complex and often chaotic in nature, making prediction incredibly challenging: we cannot expect to make perfect predictions all of the time. Instead, we look for specific states of the system that lead to more predictable behavior than others, often termed “forecasts of opportunity.” When these opportunities are not present, scientists need prediction systems that are capable of saying “I don't know.” We introduce a novel loss function, termed the “NotWrong loss,” that allows neural networks to identify forecasts of opportunity for classification problems. The NotWrong loss introduces an abstention class that allows the network to identify the more confident samples and abstain (say “I don't know”) on the less confident samples. The abstention loss is designed to abstain on a user‐defined fraction of the samples via a standard adaptive controller. Unlike many machine learning methods used to reject samples post‐training, the NotWrong loss is applied during training to preferentially learn from the more confident samples. We show that the NotWrong loss outperforms other existing loss functions for multiple climate use cases. The implementation of the proposed loss function is straightforward in most network architectures designed for classification as it only requires the addition of an abstention class to the output layer and modification of the loss function.

Controlled abstention neural networks for identifying skillful predictions for regression problems

“…Note that Eq. 4 is very similar to the abstention loss of Thulasidasan (2020, Chapter 4) and Barnes and Barnes (2021) for classification networks.…”

On the performance of uncertainty estimation methods for deep-learning based image classification models

“…In this initial study, we restrict attention to UE methods that can be computed deterministically from any of the model's ouputs or intermediate signals, without depending on the available data. This includes the conventional softmax response (also known as maximum class probability, MCP) and the entropy of the softmax output, for all models, as well as mutual information, predictive variance and mean variance, computed from the multiple realizations of the model's softmax output, in the case of ensemble models-but excludes any method that trains an auxiliary model to provide UE, such as [Geifman and El-Yaniv 2019, DeVries and Taylor 2018, Barnes and Barnes 2021, Corbière et al 2021. We focus on the task of selective classification [Geifman and El-Yaniv 2017] and, following [Galil et al 2022], evaluate all methods using the RC curve.…”