Universal Batch Learning with Log-Loss

Fogel, Yaniv; Feder, Meir

doi:10.1109/isit.2018.8437543

Cited by 15 publications

(14 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Theorem 1 (Fogel and Feder (2018)). The universal learner, denoted as the pNML, minimizes the regret for the worst case test label…”

Section: Notation and Preliminariesmentioning

confidence: 96%

“…The pNML learner. The pNML learner is the min-max solution of the supervised batch learning in the individual setting (Fogel and Feder, 2018). For sequential prediction it is termed the conditional normalized maximum likelihood (Rissanen and Roos, 2007;Roos and Rissanen, 2008).…”

Section: Dnn Adaptationmentioning

confidence: 99%

“…An additional assumption required to solve the problem is related to how the data and the labels are generated. In this work we consider the individual setting (Merhav and Feder, 1998;Fogel and Feder, 2018;Bibas et al, 2019b,a), where the data and labels, both in the training and test, are specific individual quantities: We do not assume any probabilistic relationship between them, the labels may even be assigned in an adversarial manner.…”

Section: Notation and Preliminariesmentioning

confidence: 99%

“…If the true label is one of those outcomes it will lead to a higher regret. For more information see Fogel and Feder (2018).…”

Section: Notation and Preliminariesmentioning

confidence: 99%

“…The pNML learner (Fogel and Feder, 2018) was proposed as the min-max solution of the regret, where the minimum is over the learner choice and the maximum is for any possible test label value. Intuitively, the pNML assigns a probability for a potential outcome as follows: Add the test sample to the training set with an arbitrary label, find the ERM solution of this new set, and take the probability it gives to the assumed label.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection

Bibas¹,

Feder²,

Hassner³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Detecting out-of-distribution (OOD) samples is vital for developing machine learning based models for critical safety systems. Common approaches for OOD detection assume access to some OOD samples during training which may not be available in a real-life scenario. Instead, we utilize the predictive normalized maximum likelihood (pNML) learner, in which no assumptions are made on the tested input. We derive an explicit expression of the pNML and its generalization error, denoted as the regret, for a single layer neural network (NN). We show that this learner generalizes well when (i) the test vector resides in a subspace spanned by the eigenvectors associated with the large eigenvalues of the empirical correlation matrix of the training data, or (ii) the test sample is far from the decision boundary. Furthermore, we describe how to efficiently apply the derived pNML regret to any pretrained deep NN, by employing the explicit pNML for the last layer, followed by the softmax function. Applying the derived regret to deep NN requires neither additional tunable parameters nor extra data. We extensively evaluate our approach on 74 OOD detection benchmarks using DenseNet-100, ResNet-34, and WideResNet-40 models trained with CIFAR-100, CIFAR-10, SVHN, and ImageNet-30 showing a significant improvement of up to 15.6% over recent leading methods.

show abstract

“…Theorem 1 (Fogel and Feder (2018)). The universal learner, denoted as the pNML, minimizes the regret for the worst case test label…”

Section: Notation and Preliminariesmentioning

confidence: 96%

Section: Dnn Adaptationmentioning

confidence: 99%

Section: Notation and Preliminariesmentioning

confidence: 99%

“…If the true label is one of those outcomes it will lead to a higher regret. For more information see Fogel and Feder (2018).…”

Section: Notation and Preliminariesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection

Bibas¹,

Feder²,

Hassner³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Combining Batch and Online Prediction

Fogel,

Feder

2024

2024 IEEE International Symposium on Information Theory Workshops (ISIT-W)

View full text Add to dashboard Cite

Universal Learning of Individual Data

Fogel

Feder

2019

2019 IEEE International Symposium on Information Theory (ISIT)

Self Cite

View full text Add to dashboard Cite

Universal supervised learning is considered from an information theoretic point of view following the universal prediction approach, see Merhav and Feder (1998). We consider the standard supervised "batch" learning where prediction is done on a test sample once the entire training data is observed, and the individual setting where the features and labels, both in the training and test, are specific individual quantities. The information theoretic approach naturally uses the self-information loss or log-loss.Our results provide universal learning schemes that compete with a "genie" (or reference) that knows the true test label. In particular, it is demonstrated that the main proposed scheme, termed Predictive Normalized Maximum Likelihood (pNML), is a robust learning solution that outperforms the current leading approach based on Empirical Risk Minimization (ERM). Furthermore, the pNML construction provides a pointwise indication for the learnability of the specific test challenge with the given training examples. The Model ClassThe model class definition plays an important role in all settings we consider. Specifically, a model class is a set of conditional probability distributionswhere Θ is a general index set. This is equivalent to saying that there is a set of stochastic functions {y = g θ (x), θ ∈ Θ} used to explain the relation between x and y.A major issue is how to choose a model class. As common sense indicates, on one hand one may wish to choose a large as possible class, so that any possible relation between x and y can be captured by some member in the class. However, if the class is too large, it may not be "learnable". That is, it will be impossible to deduce reliably on the large class based on the finite training example of size N . This notion appears in classical statistical reasoning and expressed, e.g., as the bias-variance trade-off. This major issue of choosing the model class will be discussed briefly towards the end of the paper, but throughout the paper we assume that P Θ is given.

show abstract

Universal Batch Learning with Log-Loss

Cited by 15 publications

References 10 publications

Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection

Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection

Combining Batch and Online Prediction

Universal Learning of Individual Data

Contact Info

Product

Resources

About