Comparing Correction Methods to Reduce Misclassification Bias

Kloos, Kevin; Meertens, Quinten; Scholtus, Sander; Karch, Julian D.

doi:10.1007/978-3-030-76640-5_5

Cited by 5 publications

(26 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The assumptions made in the experimental phase are considered to be the most restrictive ones. The answer to the key question under those restrictive assumptions has been provided by Kloos et al (2020) and it is rather conclusive. A drawback of their result is that data are assumed to be annotated manually in each time period.…”

Section: Methodsmentioning

confidence: 99%

“…The setup corresponds to the double sampling scheme introduced by Tenenbein (1970). Among the correction methods discussed by Buonaccorsi (2010), the so-called calibration estimator then outperforms all the others in terms of mean squared error, as proved theoretically by Kloos et al (2020).…”

Section: Introductionmentioning

confidence: 94%

“…A wide range of correction methods to reduce misclassification bias is available, see Buonaccorsi (2010). As briefly indicated in Section 1, Kloos et al (2020) compared several correction methods aimed at improving the accuracy of estimators for α. Two correction methods were most promising.…”

Section: The Experimental Phasementioning

confidence: 99%

“…in which C is the column-normalized confusion matrix obtained from the test set, i.e., with entries c ab = n ab /n +b , where n +b denotes n ab + n bb . Kloos et al (2020) have shown that if the test set is indeed a random sample from the target population, then the mean squared error of α c is always smaller than that of α p .…”

Section: The Experimental Phasementioning

confidence: 99%

“…In this paper, we aim to prove which of the two popular correction methods discussed by Buonaccorsi (2010) reduces the mean squared error of statistical output most, under a specific type of concept drift known as prior probability shift (Moreno-Torres et al, 2012). Our paper deliberately focuses on the production process (where concept drift arises), building on the results obtained by Kloos et al (2020) for the preceding experimental phase. Our numerical analyses will show, for the first time, that a decision boundary arises.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Improving the output quality of official statistics based on machine learning algorithms

Meertens¹,

Diks²,

Herik³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

National statistical institutes currently investigate how to improve the output quality of official statistics based on machine learning algorithms. A key obstacle is concept drift, i.e., when the joint distribution of independent variables and a dependent (categorical) variable changes over time. Under concept drift, a statistical model requires regular updating to prevent it from becoming biased. However, updating a model asks for additional data, which are not always available. In the literature, we find a variety of bias correction methods as a promising solution.In the paper, we will compare two popular correction methods: the misclassification estimator and the calibration estimator. For prior probability shift (a specific type of concept drift), we investigate the two correction methods theoretically as well as experimentally. Our theoretical results are expressions for the bias and variance of both methods. As experimental result, we present a decision boundary (as a function of (a) model accuracy, (b) class distribution and (c) test set size) for the relative performance of the two methods. Close inspection of the results will provide a deep insight into the effect of prior probability shift on output quality, leading to practical recommendations on the use of machine learning algorithms in official statistics. ‡

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 94%

Section: The Experimental Phasementioning

confidence: 99%

Section: The Experimental Phasementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Improving the output quality of official statistics based on machine learning algorithms

Meertens¹,

Diks²,

Herik³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Methods for Estimating the Quality of Multisource Statistics

Delden¹,

Scholtus²,

Waal³

et al. 2023

Advances in Business Statistics, Methods and Data Collection

View full text Add to dashboard Cite

show abstract

Misclassification Bias in Computational Social Science: A Simulation Approach for Assessing the Impact of Classification Errors on Social Indicators Research

Smetanin

Komarov

2022

IEEE Access

View full text Add to dashboard Cite

A growing body of literature has examined the potential of machine learning algorithms in constructing social indicators based on the automatic classification of digital traces. However, as long as the classification algorithms' predictions are not completely error-free, the estimate of the relative occurrence of a particular class may be affected by misclassification bias, thereby affecting the value of the calculated social indicator. Although a significant amount of studies have investigated misclassification bias correction techniques, they commonly rely on a set of assumptions that are likely to be violated in practice, which calls into question the effectiveness of these methods. Thus, there is a knowledge gap with respect to the assessment of misclassification bias's impact on a specific social indicator formula without strict reference to the number of classes. Moreover, given the erroneous nature of automatic classification algorithms, the quality of a predicted indicator can be assessed not only using regression quality metrics, as was done in existing literature, but also using correlation metrics. In this paper, we propose a simulation approach for assessing the impact of misclassification bias on the calculated social indicators in terms of regression and correlation metrics. The proposed approach focuses on indicators calculated based on the distribution of classes and can process any number of classes. The proposed approach allows selecting the most appropriate classification model for a particular social indicator, and vice versa. Moreover, it allows for assessment of the optimistic level of correlation between the indicator calculated based on the results of the classification algorithm and the true underlying indicator.

show abstract

Comparing Correction Methods to Reduce Misclassification Bias

Cited by 5 publications

References 25 publications

Improving the output quality of official statistics based on machine learning algorithms

Improving the output quality of official statistics based on machine learning algorithms

Methods for Estimating the Quality of Multisource Statistics

Misclassification Bias in Computational Social Science: A Simulation Approach for Assessing the Impact of Classification Errors on Social Indicators Research

Contact Info

Product

Resources

About