Preventing bias from selective non-response in population-based survey studies: findings from a Monte Carlo simulation study

Abstract

Background Health researchers often use survey studies to examine associations between risk factors at one time point and health outcomes later in life. Previous studies have shown that missing not at random (MNAR) may produce biased estimates in such studies. Medical researchers typically do not employ statistical methods for treating MNAR. Hence, there is a need to increase knowledge about how to prevent occurrence of such bias in the first place. Methods Monte Carlo simulations were used to examine the degree to which selective non-response leads to biased estimates of associations between risk factors and health outcomes when persons with the highest levels of health problems are under-represented or totally missing from the sample. This was examined under different response rates and different degrees of dependency between non-response and study variables. Results Response rate per se had little effect on bias. When extreme values on the health outcome were completely missing, rather than under-represented, results were heavily biased even at a 70% response rate. In most situations, 50–100% of this bias could be prevented by including some persons with extreme scores on the health outcome in the sample, even when these persons were under-represented. When some extreme scores were present, estimates of associations were unbiased in several situations, only mildly biased in other situations, and became biased only when non-response was related to both risk factor and health outcome to substantial degrees. Conclusions The potential for preventing bias by including some extreme scorers in the sample is high (50–100% in many scenarios). Estimates may then be relatively unbiased in many situations, also at low response rates. Hence, researchers should prioritize to spend their resources on recruiting and retaining at least some individuals with extreme levels of health problems, rather than to obtain very high response rates from people who typically respond to survey studies. This may contribute to preventing bias due to selective non-response in longitudinal studies of risk factors and health outcomes.

A framework for exploring non-response patterns over time in health surveys

“…Inviting more people from less-responsive subgroups will directly result in a lower overall response, however, this does not necessarily introduce bias. On the contrary, a modelling study showed that by including at least some persons in upper and lower percentiles of the distributions of outcomes reduced the risk of bias in associations between determinants and outcomes [ 11 ]. Recruiting persons from the extremes of the distribution likely increases the variation in the study sample.…”

“…Correcting for the latter is not possible, although it has been suggested that a low response rate will not bias associations in studies with large samples sizes and sufficient variation in the outcome of interest [ 9 , 10 ]. This suggestion was supported by a modelling study showing that estimates of associations between determinants and outcomes were relatively unbiased in scenarios where variation was maintained by including some individuals with outcomes at extreme values of the distribution [ 11 ].…”

A Common Family Factor Underlying Language Difficulties and Internalizing Problems: Findings From a Population-Based Sibling Study

“…The model assumes a latent continuous normally distributed variable underlying the observed categorical variable. Standardized output in Mplus gives estimates of the association between a predictor and this latent normally distributed variable and can be interpreted in the same way as standardized estimates from linear models ( Gustavson et al, 2019 ; Muthén & Muthén, 2009 ). Chi-square values obtained from this estimator cannot be used directly for testing differences of model fit, and the DIFFTEST option was therefore used ( Muthén, 2010 ).…”

Association of attrition with mortality: findings from 11 waves over three decades of the Whitehall II study

“…This implies that internal and external validity of studies may be affected in certain circumstances. 4 33 34 For example, selection can lead to collider bias (a bias occurring when two variables independently affect a third variable, and that third variable is conditioned upon), which can bias estimations. 4 Complete case analysis would not be problematic if it can be assumed that missingness occurs completely at random.…”