Calibrating Deep Neural Networks using Focal Loss

Abstract

Miscalibration -a mismatch between a model's confidence and its correctness -of Deep Neural Networks (DNNs) makes their predictions hard to rely on. Ideally, we want networks to be accurate, calibrated and confident. We show that, as opposed to the standard cross-entropy loss, focal loss Lin et al. [2017] allows us to learn models that are already very well calibrated. When combined with temperature scaling, whilst preserving accuracy, it yields state-of-the-art calibrated models. We provide a thorough analysis of the factors causing miscalibration, and use the insights we glean from this to justify the empirically excellent performance of focal loss. To facilitate the use of focal loss in practice, we also provide a principled approach to automatically select the hyperparameter involved in the loss function. We perform extensive experiments on a variety of computer vision and NLP datasets, and with a wide variety of network architectures, and show that our approach achieves state-of-the-art accuracy and calibration in almost all cases.

Mining Minority-class Examples With Uncertainty Estimates

“…As explained earlier, this problem occurs due to two main reasons: overfitting in the final softmax layer and subdued tail-class activations. The overfitting and the resulting miscalibration in softmax probabilities is a known phenomenon observed in many modern multi-class classification networks [12,20]. This has been attributed to prolonged training to minimize the negative log-likelihood loss on a network with a large capacity.…”

“…To mitigate these problems, the overfit softmax layer of the model f θ is replaced with a structurally similar newly-initialized layer (recalibration layer). The intuition behind replacing only the last layer is based on the idea that overfitting and miscalibration are mainly attributed to weight magnification particularly in the last layer of the neural network [20]. The new layer is trained with early stopping and focal loss which help solve the problem of overfitting and subdued activations, respectively.…”

“…Focal loss, which is a dynamically weighted variant of the negative log-likelihood loss, helps the network to focus on statically-rare tail-class examples, and hence mitigate the problem of subdued activations. Additionally, focal loss also acts as an entropy-maximizing regularisation which has an added benefit further preventing the model to become overconfident on head-classes [20].…”

“…Second, even in the output space, the bias in the trained model due to class imbalance subdues tail-class activations, resulting in distortion of the model's uncertainty estimates. This effect is aggravated in deep neural networks where due to pervasive overfitting of the final softmax layer [12,20], head-class activations completely eclipse tail-class activations. Hence, the otherwise successful informative-sample mining methods in active learning literature, which rely on the model's uncertainties in the output space, are well-known to perform poorly in mining tail-class examples [1,2,3,6].…”

Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition

“…However, this raises a new concern, to what extend the network's predictions are likely to be correct? As these deep network tries to reduce the negative log-likelihood loss, they overfit to datasets, rendering its predictions to be over-confident and less trustworthy (Mukhoti et al, 2020). Here, the network is termed to be poorly calibrated.…”