The Hitchhiker’s Guide to Prior-Shift Adaptation

Abstract

In many computer vision classification tasks, class priors at test time often differ from priors on the training set. In the case of such prior shift, classifiers must be adapted correspondingly to maintain close to optimal performance. This paper analyzes methods for adaptation of probabilistic classifiers to new priors and for estimating new priors on an unlabeled test set. We propose a novel method to address a known issue of prior estimation methods based on confusion matrices, where inconsistent estimates of decision probabilities and confusion matrices lead to negative values in the estimated priors. Experiments on fine-grained image classification datasets provide insight into the best practice of prior shift estimation and classifier adaptation, and show that the proposed method achieves state-of-the-art results in prior adaptation. Applying the best practice to two tasks with naturally imbalanced priors, learning from webcrawled images and plant species classification, increased the recognition accuracy by 1.1% and 3.4% respectively.

Plant recognition by AI: Deep neural nets, transformers, and kNN in deep embeddings

“…The results show that in all cases, prior shift adaptation improves the recognition accuracy. The EM algorithm of Saerens et al ( 2002 ) achieves the best result in three cases, the CM-L method of Sipka et al ( 2022 ) in one case, but the differences are very small among the three compared prior shift adaptation methods.…”

“…We test existing prior shift adaptation methods and their impact on classification accuracy. The experiments with state-of-theart methods for prior shift estimation (Saerens et al, 2002;Sipka et al, 2022), evaluated in Table 6, show that all three compared methods improve the classification accuracy in all cases. The differences among all three methods are rather small, EM achieving slightly better results in 3 of 4 cases.…”

“…The evaluation and test set priors p e ( c k ) are, however, unknown. To evaluate the impact of changing class priors, we compare three existing prior estimation algorithms—the Expectation–maximization algorithm (EM) of Saerens et al ( 2002 ) and the recently proposed CM-L and SCM-L methods of Sipka et al ( 2022 ).…”

On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey

“…The model tries to approximate the true class priors p(y|x) by learning from more representative training data or using different optimization techniques. However, in practical evaluation scenarios, the prior probabilities may differ from that of the training set and may even change from one domain to another, which is often called prior shift or label shift [141].…”

“…where the ratio p(y) p(y) implies the prior shift. In [141], authors propose the test-time adaptation of a fixed pretrained classifier after a prior shift happens by re-weighting its predictions based on confusion matrices. To avoid over-confident predictions due to overfitting to some classes, [142] propose to calibrate the confidence of classifier predictions by adding class-specific bias terms:…”