How well do covariates perform when adjusting for sampling bias in online COVID-19 research? Insights from multiverse analyses

Abstract

COVID-19 research has relied heavily on convenience-based samples, which—though often necessary—are susceptible to important sampling biases. We begin with a theoretical overview and introduction to the dynamics that underlie sampling bias. We then empirically examine sampling bias in online COVID-19 surveys and evaluate the degree to which common statistical adjustments for demographic covariates successfully attenuate such bias. This registered study analysed responses to identical questions from three convenience and three largely representative samples (total N = 13,731) collected online in Canada within the International COVID-19 Awareness and Responses Evaluation Study ( www.icarestudy.com ). We compared samples on 11 behavioural and psychological outcomes (e.g., adherence to COVID-19 prevention measures, vaccine intentions) across three time points and employed multiverse-style analyses to examine how 512 combinations of demographic covariates (e.g., sex, age, education, income, ethnicity) impacted sampling discrepancies on these outcomes. Significant discrepancies emerged between samples on 73% of outcomes. Participants in the convenience samples held more positive thoughts towards and engaged in more COVID-19 prevention behaviours. Covariates attenuated sampling differences in only 55% of cases and increased differences in 45%. No covariate performed reliably well. Our results suggest that online convenience samples may display more positive dispositions towards COVID-19 prevention behaviours being studied than would samples drawn using more representative means. Adjusting results for demographic covariates frequently increased rather than decreased bias, suggesting that researchers should be cautious when interpreting adjusted findings. Using multiverse-style analyses as extended sensitivity analyses is recommended. Supplementary Information The online version contains supplementary material available at 10.1007/s10654-022-00932-y.

Assessment and determinants of depression and anxiety on a global sample of sexual and gender diverse people at high risk of HIV: a public health approach

“…The degree and direction in which the selection bias of internet convenience sampling may under or overestimate the relationship between sample characteristics and measured outcomes are difficult to predict and control [ 3 , 71 ]. We identified and included demographic covariates associated with the potential bias described above to reduce but not eliminate the potential bias [ 72 – 74 ]. The survey intended to include the different sexual and gender diverse people.…”

Appealing to motivation to change attitudes, intentions, and behavior: A systematic review and meta-analysis of 702 experimental tests of the effects of motivational message matching on persuasion.

“…Further, the operational factors covered in our review show many substantial associations with one another, with certain attributes even nearly perfectly co-occurring (e.g., 97% of effects following a single-exposure message were assessed on the day of exposure to the message). This type of imbalance makes interpreting moderation difficult, and adjusting for the influence of confounders using a purely analytical approach is unlikely to be sufficient (and can even risk increasing bias in causal estimation; Hernán et al, 2004; Joyal-Desmarais et al, in press; Schisterman et al, 2009). Consequently, design-based methods, such as carefully designed experimental studies, are recommended to examine the influence of these operational factors in the future.…”

Canadian healthcare workers’ mental health and health behaviours during the COVID-19 pandemic: results from nine representative samples between April 2020 and February 2022

“…While representative sampling has strengths (e.g., proportion of the population by sociodemographic characteristics, randomdigit sampling), it is not without selection bias. As previously shown (Joyal-Desmarais et al, 2022), the samples from the Leger panel were not fully representative of the Canadian population (e.g., age, language, education). Also, due to the sampling methods of the Leger Opinion polling firm, it is not possible to know whether a respondent has participated in and contributed to several data collection times.…”