Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression

Abstract

Heteroscedastic regression is the task of supervised learning where each label is subject to noise from a different distribution. This noise can be caused by the labelling process, and impacts negatively the performance of the learning algorithm as it violates the i.i.d. assumptions. In many situations however, the labelling process is able to estimate the variance of such distribution for each label, which can be used as an additional information to mitigate this impact. We adapt an inverse-variance weighted mean square error, based on the Gauss-Markov theorem, for parameter optimization on neural networks. We introduce Batch Inverse-Variance, a loss function which is robust to near-ground truth samples, and allows to control the effective learning rate. Our experimental results show that BIV improves significantly the performance of the networks on two noisy datasets, compared to L2 loss, inverse-variance weighting, as well as a filteringbased baseline.

Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation

“…Batch Inverse-Variance (BIV) weighting (Mai et al, 2021) leverages the additional information σ 2 k , which is assumed to be provided, to learn faster and obtain better performance in the case of heteroscedastic noise on the labels. Applied to L2 loss, it optimizes the neural network parameters θ using the following loss function for a mini-batch D of size K 2 :…”

“…As explained in Mai et al (2021), ξ is a hyperparameter which is important for the stability of the optimization process. A higher ξ limits the highest weights, thus preventing very small variance samples from dominating the loss function for a mini-batch.…”

“…The study on the variance allows a better understanding of the advantage of the BIV weights (Mai et al, 2021) compared to the UWAC weights (Wu et al, 2021) and SUNRISE weights (Lee et al, 2020).…”

“…We merge the predicted variances of several variance networks through a mixture of Gaussians, which has been shown to be a very reliable method to capture predictive uncertainty (Ovadia et al, 2019). We then use Batch Inverse-Variance (BIV) (Mai et al, 2021), which has been shown to significantly improve the performance of supervised learning with neural networks in the case of heteroscedastic regression. BIV is normalized, which makes it ideal to cope with different and time-varying scales of variance.…”

“…We replaced inMai et al (2021) with ξ to avoid confusion with the reinforcement learning conventions.…”