Evaluating the impact of response enhancement methods on the risk of nonresponse bias and survey costs

R-indicators are increasingly used as nonresponse bias indicators. However, their effectiveness depends on the auxiliary data used in their estimation. Because of this, it is not always clear for practitioners what the magnitude of the R-indicator implies for bias in other survey variables, or how adjustment on auxiliary variables will affect it. In this article, we investigate these potential limitations of R-indicators in a case study using data from the Swiss European Social Survey (ESS5), which included a nonresponse follow-up (NRFU) survey. First, we analyse correlations between estimated response propensities based on auxiliary data from the register-based sampling frame, and responses to survey questions also included in the NRFU. We then examine how these relate to bias detected by the NRFU, before and after adjustment, and to predictions of the risk of bias provided by the R-indicator. While the results lend support for the utility of R-indicators as summary statistics of bias risk, they suggest a need for caution in their interpretation. Even where auxiliary variables are correlated with target variables, more bias in the former (resulting in a larger R-indicator) does not automatically imply more bias in the latter, nor does adjustment on the former necessarily reduce bias in the latter.

Section: Discussionmentioning

confidence: 99%

Section: Datamentioning

confidence: 99%

Section: Estimating Response Propensitiesmentioning

confidence: 99%

See 1 more Smart Citation

A Validation of R-Indicators as a Measure of the Risk of Bias using Data from a Nonresponse Follow-Up Survey

Roberts

Vandenplas²,

Herzing

2020

“…Researchers have used R-indicators to assess the extent to which a net sample is representative of the target population or a gross sample. For instance, data of recruited samples have been compared with census, administrative, or population register data (e.g., Moore et al 2016;Schouten et al 2012;Luiten and Schouten 2013;Roberts et al 2014). R-indicators can also be used as indicators for representativeness in panel studies (Schouten et al 2012).…”

Section: Analytical Approachmentioning

confidence: 99%

Evolution of the Initially Recruited SHARE Panel Sample Over the First Six Waves

Friedel

Birkenbach

2020

Attrition is a frequently observed phenomenon in panel studies. The loss of panel members over time can hamper the analysis of panel survey data. Based on data from the Survey of Health, Ageing and Retirement in Europe (SHARE), this study investigates changes in the composition of the initially recruited first-wave sample in a multi-national face-to-face panel survey of an older population over waves. By inspecting retention rates and R-indicators, we found that, despite declining retention rates, the composition of the initially recruited panel sample in Wave 1 remained stable after the second wave. Thus, after the second wave there is no further large decline in representativeness with regard to the first wave sample. Changes in the composition of the sample after the second wave over time were due mainly to mortality-related attrition. Non-mortality-related attrition had a slight effect on the changes in sample composition with regard to birth in survey country, area of residence, education, and social activities. Our study encourages researchers to investigate further the impact of mortality- and non-mortality-related attrition in multi-national surveys of older populations.

“…Other examples use only data from the past implementations of a survey to estimate coefficients in the response propensity model, which are then applied to covariates during the in-progress data collection period (Peytchev et al 2010;Roberts et al 2014;Calinescu et al 2013;Jackson et al 2020). Unlike the prior examples, these applications do not consider information learned during the in-progress data collection period.…”

Section: Introductionmentioning

confidence: 99%

Predicting Days to Respondent Contact in Cross-Sectional Surveys Using a Bayesian Approach

Coffey,

Elliott

2023

Surveys estimate and monitor a variety of data collection parameters, including response propensity, number of contacts, and data collection costs. These parameters can be used as inputs to a responsive/adaptive design or to monitor the progression of a data collection period against predefined expectations. Recently, Bayesian methods have emerged as a method for combining historical information or external data with data from the in-progress data collection period to improve prediction. We develop a Bayesian method for predicting a measure of case-level progress or productivity, the estimated time lag, in days, between first contact attempt and first respondent contact. We compare the quality of predictions from the Bayesian method to predictions generated from more commonly-used predictive methods that leverage data from only historical data collection periods or the in-progress round of data collection. Using prediction error and misclassification as short- or long- day lags, we demonstrate that the Bayesian method results in improved predictions close to the day of the first contact attempt, when these predictions may be most informative for interventions or interviewer feedback. This application adds to evidence that combining historical and current information about data collection, in a Bayesian framework, can improve predictions of data collection parameters.