Treating Nonresponse in Probability-Based Online Panels through Calibration: Empirical Evidence from a Survey of Political Decision-Making Procedures

Abstract

The use of probability-based panels that collect data via online or mixed-mode surveys has increased in the last few years as an answer to the growing concern with the quality of the data obtained with traditional survey modes. However, in order to adequately represent the general population, these tools must address the same sources of bias that affect other survey-based designs: namely under coverage and non-response. In this work, we test several approaches to produce calibration estimators that are suitable for survey data affected by non response where auxiliary information exists at both the panel level and the population level. The first approach adjusts the results obtained in the cross-sectional survey to the population totals, while, in the second, the weights are the result of two-step process where different adjusts on the sample, panel, and population are done. A simulation on the properties of these estimators is performed. In light of theory and simulation results, we conclude that weighting by calibration is an effective technique for the treatment of non-response bias when the response mechanism is missing at random. These techniques have also been applied to real data from the survey Andalusian Citizen Preferences for Political Decision-Making Procedures.

Calibration and XGBoost reweighting to reduce coverage and non-response biases in overlapping panel surveys: Application to the Healthcare and Social Survey

“…Thus, to obtain the cross-sectional estimator for the study variables of each ESSA measurement, we start from the H-T estimator (1) adjusted for non-response (7), combined from the panel and new samples (9) and finally calibrate to increase the representativeness of the sample (10). This estimator serves as the basis for calculating the absolute (13) and relative (14) change estimators between measurement j and 1, and for obtaining the different estimators to measure the absolute and relative gender gap in a given measurement (15 and 16).…”

“…They consider three possible calibration approaches: initial calibration (at the beginning of the panel, the weights of the units in the panel are calibrated), final calibration (at measurement t the weights of the individuals in the sample are adjusted by calibration) and initial and subsequent final calibration (both initial and final calibration are carried out). Several approaches are tested in [10] to produce calibration estimators which are suitable for survey data affected by non response where auxiliary information exists at both the panel and population level.…”

Calibration Estimation of Population Proportion in Probability Proportional to Size Sampling in the Presence of Non-Response

“…Several approaches were explored to develop calibration estimators that were suitable for survey data affected by nonresponse or missing data, where auxiliary variables were available at both the population level and the panel level (6) . The popular calibration estimator of population total in PPS sampling was suggested when the auxiliary information is assumed to be known (7) .…”

Calibration and XGBoost reweighting to reduce coverage and non-response biases in overlapping panel surveys: application to the Healthcare and Social Survey

“…They consider three possible calibration approaches: initial calibration (at the beginning of the panel, the weights of the units in the panel are calibrated), final calibration (at measurement t the weights of the individuals in the sample are adjusted by calibration) and initial and subsequent final calibration (both initial and final calibration are carried out). Several approaches are tested in [15] to produce calibration estimators which are suitable for survey data affected by non response where auxiliary information exists at both the panel and population level.…”