Inference from Non-Probability Surveys with Statistical Matching and Propensity Score Adjustment Using Modern Prediction Techniques

Abstract

Online surveys are increasingly common in social and health studies, as they provide fast and inexpensive results in comparison to traditional ones. However, these surveys often work with biased samples, as the data collection is often non-probabilistic because of the lack of internet coverage in certain population groups and the self-selection procedure that many online surveys rely on. Some procedures have been proposed to mitigate the bias, such as propensity score adjustment (PSA) and statistical matching. In PSA, propensity to participate in a nonprobability survey is estimated using a probability reference survey, and then used to obtain weighted estimates. In statistical matching, the nonprobability sample is used to train models to predict the values of the target variable, and the predictions of the models for the probability sample can be used to estimate population values. In this study, both methods are compared using three datasets to simulate pseudopopulations from which nonprobability and probability samples are drawn and used to estimate population parameters. In addition, the study compares the use of linear models and Machine Learning prediction algorithms in propensity estimation in PSA and predictive modeling in Statistical Matching. The results show that statistical matching outperforms PSA in terms of bias reduction and Root Mean Square Error (RMSE), and that simpler prediction models, such as linear and k-Nearest Neighbors, provide better outcomes than bagging algorithms.

Self-Perceived Health, Life Satisfaction and Related Factors among Healthcare Professionals and the General Population: Analysis of an Online Survey, with Propensity Score Adjustment

“…As stated in Section 1, it is known that PSA can reduce the selection bias at the cost of increasing the variance because of the complexity added by the predictive models. However, the bias-variance trade-off is often positive, as the mean square error gets reduced after the application of PSA in certain situations, according to literature [11,14,15,23,24].…”

“…Another goal of this research was to explore the use of machine learning (ML) classification algorithms to remove selection bias by reweighting the study variables via PSA. ML techniques are commonly employed in epidemiology [17][18][19], and statistical algorithms have been used to weight variables in recent health surveys [20][21][22].These techniques have also shown good properties in simulated data in terms of bias reduction [23,24] but at the cost of increasing the variance of the estimates. However, the mean square error (MSE), which combines bias and variance, is reduced with PSA in some situations, meaning that its application can be recommended in nonprobability sampling contexts.…”

Estimating General Parameters from Non-Probability Surveys Using Propensity Score Adjustment

“…We used parametric methods to obtain the estimated propensities but we could use machine learning techniques as regression trees, spline regression, random forests etc. Recently [24,29] presented simulation studies where decision trees, k-nearest neighbors, Naive Bayes, Random Forest, Gradient Boosting Machine and Model Averaged Neural Networks are used for propensity score estimation. These studies compare the empirical efficiency of the use of linear models and Machine Learning prediction algorithms in estimation of linear parameters, but the theory is more complex and has not yet been developed.…”

“…The choice of prediction models has been studied in literature; the usual method is linear regression but other approaches such as donor imputation [13] or Machine Learning algorithms [19,29] have been listed as alternatives. Under certain conditions, Statistical Matching can reduce bias and mean square error to a greater extent than PSA [29].…”

Weight smoothing for nonprobability surveys

“…The predictive model is fitted using data from the nonprobability sample. Statistical matching has also been proven to mitigate selection bias in nonprobability samples (Castro-Martín et al 2020). The combination of both strategies via doubly robust estimators may outperform both approaches on their own Chen et al (2020).…”