The impact of survey mode on US national estimates of adolescent drug prevalence: results from a randomized controlled study

Abstract

Background and Aims Increasing numbers of school‐based drug surveys are transitioning data collection to electronic tablets from paper‐and‐pencil, which may produce a survey mode effect and consequent discontinuity in time trends for population estimates of drug prevalence. This study tested whether (a) overall, self‐reported drug use prevalence is higher on electronic tablets versus paper‐and‐pencil surveys, (b) socio‐demographics moderate survey mode effects and (c) levels of missing data are lower for electronic tablet versus paper‐and‐pencil modes. Design A randomized controlled experiment. Setting Results are nationally representative of students in the contiguous United States. Participants A total of 41 866 8th, 10th and 12th grade students who participated in the 2019 Monitoring the Future school‐based survey administration. Intervention and comparator Surveys were administered to students in a randomly selected half of schools with electronic tablets (intervention) and with paper‐and‐pencil format (comparator) for the other half. Measurements Primary outcome was the total number of positive drug use responses. Secondary outcomes were the percentage of respondents completing all drug questions, percentage of drug questions unanswered and mean number of missing drug items. Findings The relative risk (RR) for total number of positive drug use responses for electronic tablets versus paper‐and‐pencil surveys were small and their 95% confidence intervals (CI) included the value of one for reporting intervals of life‐time (RR = 1.03; 95% CI, 0.93–1.14), past 12 months (RR = 1.01; 95% CI, 0.91–1.11), past 30 days (RR = 1.05; 95% CI, 0.93–1.20) and for heavy use (RR = 1.10; 95% CI, 0.93–1.29). Multiplicative interaction tests indicated no moderation of these relative risks by race (white versus non‐white), population density, census region, public/private school, year of school participation, survey version or non‐complete drug responses. Levels of missing data were significantly lower for electronic tablets versus paper‐and‐pencil surveys. Conclusions Adolescent drug prevalence estimates in the United States differed little across electronic tablet versus paper‐and‐pencil survey modes, and showed little to no effect modification by socio‐demographics. Levels of missing data were lower for electronic tablets.

Tobacco, Alcohol, Cannabis, and Other Drug Use in the US Before and During the Early Phase of the COVID-19 Pandemic

“…A US Centers for Disease Control and Prevention web-based survey suggested that many adults used substances to cope with COVID-19–related stress, but no corresponding information was provided about possible reductions in substance use . Most recently, Monitoring the Future data from students queried in January to June 2021 showed broad declines across nearly all substances for 8th, 10th, and 12th grade students (approximately ages 13-14, 15-16, and 17-18 years), results which were consistent with our findings for teenaged participants in the PATH Study, interviewed in July to December 2020 …”

“… 14 Most recently, Monitoring the Future data from students queried in January to June 2021 showed broad declines across nearly all substances for 8th, 10th, and 12th grade students (approximately ages 13-14, 15-16, and 17-18 years), results which were consistent with our findings for teenaged participants in the PATH Study, interviewed in July to December 2020. 15 …”

Research on substance use disorders during the COVID-19 pandemic

“…Similarly, school closings during COVID-19 have jeopardized the ability of those conducting Monitoring The Future (MTF; an annual national survey on drug use conducted in school classrooms) to collect data in person. COVID-19 will likely trigger a swift switch to web-based surveys, and researchers have already initiated studies to assess comparability of results using different data collection methods ( Miech et al, 2020 ). Additionally, researchers need approaches to better cover hard-to-reach populations (e.g., incarcerated or homeless) or those without internet access, and data collection methods that are more timely and representative of the population and that combine multiple modalities.…”

Electronic Versus Paper and Pencil Survey Administration Mode Comparison: 2019 Youth Risk Behavior Survey*

“…Prior studies of survey administration mode effect have found computer‐assisted and web‐delivered administration of school‐based surveys yielded results that were largely equivalent to traditional paper‐and‐pencil survey results 2‐5 . While some studies have shown differences in prevalence estimates for less socially desirable behaviors, eg, sexual behavior and substance use, 6‐8 a clear mode effect for sensitive risk behaviors has not been consistently reproduced in other studies 9‐11 . To gain a more comprehensive understanding of the potential impact of survey administration mode, other studies have explored measures in addition to prevalence estimates, including percent nonresponse, 4 percent of specific subject questions answered, 9,12 and time to complete survey 12 .…”

“…[2][3][4][5] While some studies have shown differences in prevalence estimates for less socially desirable behaviors, eg, sexual behavior and substance use, [6][7][8] a clear mode effect for sensitive risk behaviors has not been consistently reproduced in other studies. [9][10][11] To gain a more comprehensive understanding of the potential impact of survey administration mode, other studies have explored measures in addition to prevalence estimates, including percent nonresponse, 4 percent of specific subject questions answered, 9,12 and time to complete survey. 12 Similarly, in this study, select metrics across a broad range of topic areas were examined to evaluate data quality and representativeness between survey administration modes.…”