Misclassification Bias in Computational Social Science: A Simulation Approach for Assessing the Impact of Classification Errors on Social Indicators Research

Abstract

A growing body of literature has examined the potential of machine learning algorithms in constructing social indicators based on the automatic classification of digital traces. However, as long as the classification algorithms' predictions are not completely error-free, the estimate of the relative occurrence of a particular class may be affected by misclassification bias, thereby affecting the value of the calculated social indicator. Although a significant amount of studies have investigated misclassification bias correction techniques, they commonly rely on a set of assumptions that are likely to be violated in practice, which calls into question the effectiveness of these methods. Thus, there is a knowledge gap with respect to the assessment of misclassification bias's impact on a specific social indicator formula without strict reference to the number of classes. Moreover, given the erroneous nature of automatic classification algorithms, the quality of a predicted indicator can be assessed not only using regression quality metrics, as was done in existing literature, but also using correlation metrics. In this paper, we propose a simulation approach for assessing the impact of misclassification bias on the calculated social indicators in terms of regression and correlation metrics. The proposed approach focuses on indicators calculated based on the distribution of classes and can process any number of classes. The proposed approach allows selecting the most appropriate classification model for a particular social indicator, and vice versa. Moreover, it allows for assessment of the optimistic level of correlation between the indicator calculated based on the results of the classification algorithm and the true underlying indicator.

Biases in scholarly recommender systems: impact, prevalence, and mitigation

“…Definition: The model makes assumptions to better learn the target function and to generalize the training data (Chen et al, 2020a). Impact: Strictly speaking, inductive bias is not part of the traditional definition of bias as unfairly favoring a group of people or an opinion (O'Neil, 2016;Delgado-Rodriguez & Llorca, 2004;Smetanin & Komarov, 2022). Inductive bias is considered to be positive, as it leads to more accurate results.…”

“…Bias refers to the phenomenon of unfairly favoring a group of people or an opinion (O'Neil, 2016;Delgado-Rodriguez & Llorca, 2004;Smetanin & Komarov, 2022), and is highly relevant for recommender systems. For instance, in the case of movie recommendation, users tend to rate popular movies more often, which results in unpopular movies being recommended less frequently (Park & Tuzhilin, 2008;Abdollahpouri et al, 2017).…”

Biases in Scholarly Recommender Systems: Impact, Prevalence, and Mitigation

“…Definition: The model makes assumptions to better learn the target function and to generalize the training data (Chen et al, 2020a). Impact: Strictly speaking, inductive bias is not part of the traditional definition of bias as unfairly favoring a group of people or an opinion (O'Neil, 2016;Delgado-Rodriguez and Llorca, 2004;Smetanin and Komarov, 2022). Inductive bias is considered to be positive, as it leads to more accurate results.…”

How weather impacts expressed sentiment in Russia: evidence from Odnoklassniki

“…Fourthly, the relationship between expressed sentiment and weather was observed from historical data, which might not persist into the future because of changing social and economic conditions, city environments, as well as policies ( Wang, Obradovich & Zheng, 2020 ). Lastly, the model used for extracting sentiment from posts is not completely error-free, so the estimate of the relative occurrence of a particular class may be affected by misclassification bias, thereby affecting the value of the calculated index of expressed sentiment ( Smetanin & Komarov, 2022 ).…”