Effects of Push-To-Web Mixed Mode Approaches on Survey Response Rates: Evidence from a Randomized Experiment in Emergency Departments

Abstract

Recent work involving survey administration among emergency department (ED) patients has demonstrated very low response rates using single-mode (e.g., mail-only or web-only) approaches. In this study, the authors tested several mixed-mode protocols that focus on a push-to-web approach for survey administration in this challenging but important setting. A random sample of 26,991 ED discharged-to-community (DTC) patients discharged in January 2018-March 2018 from 16 hospital-based EDs nationwide were randomized to nine survey arms, eight of which involved a web survey with one or more of the following methods of invitation to the web survey: an emailed link to the web survey, a texted link to the web survey, and/or a mailed paper invitation with information on how to access the web survey. The reference arm was standard mixed mode (mailed survey with telephone follow-up). Invitation(s) to the web survey was followed by (1) a mailed survey, (2) telephone follow-up, or (3) both. The overall response rate across all arms was 18.6% (American Association for Public Opinion Research [AAPOR] RR1). None of the tested web-survey arms had significantly higher response rates than standard mixed mode (25.5%). Protocol comparisons demonstrated that both text outreach and including a telephone mode increased response rates. When examining response by completion mode, the arms involving text invitation had a higher percentage of completed surveys that were completed by web than arms not involving text invitation. Characteristics (e.g., age) of telephone and web respondents were more representative of the sampled population than mail respondents. Older patients and male patients were significantly less likely to be reachable by email or text. These results demonstrate that sequential multi-mode approaches involving web, mail, and telephone with multiple methods of web survey invitation show the greatest promise for obtaining higher response rates and more representative respondents from the ED population.

Development and validation of a patient experience of care survey for emergency departments

“…Nonresponse bias is possible if characteristics associated with nonresponse are also associated with care experiences. 13 However, previous work suggests that adjusting for differences in case mix, as done in this study, adequately addresses any nonresponse bias. 21 Our observed response rate reflects the challenging-to-survey ED patient population that is relatively young and mobile compared to those served in other health care settings.…”

“…Previous analyses have demonstrated the need for adjustment for mode of survey administration and these case mix adjustment variables. 13,14 Response percentile captures both ED response rate and how quickly a patient responded to the survey compared to other patients in the same ED and mode of administration; 14 it is defined as the rank-ordered number of days between a respondent's discharge date and the date that data collection activities ended for the respondent relative to all eligible patients within ED and mode of administration, scaled from 0 to 1. We adjusted for this variable because later responders often provide more negative responses than those who respond quickly.…”

“…Four large‐scale experiments conducted in 2014, 2016 (two experiments), and 2018, tested the survey among patients from 12 hospitals, 50 hospitals, eight hospitals, and 16 hospitals, respectively. The primary aims of these experiments were to test mode protocols for survey administration and to test performance of survey items and modes of survey implementation; results are described elsewhere 12–15 . Item performance results informed the refinement of the survey instrument; item performance results from the 2018 experiment are described here.…”

“…The primary aims of these experiments were to test mode protocols for survey administration and to test performance of survey items and modes of survey implementation; results are described elsewhere. [12][13][14][15] Item performance results informed the refinement of the survey instrument; item performance results from the 2018 experiment are described here. Mode protocol results informed CMS' recommendation that the EDPEC DTC survey is administered in one of three modes: (1) standard mixed mode (mail with telephone follow-up), (2) web survey (by email invitation) with telephone follow-up, and (3) web survey (by email invitation) followed by a mailed survey with telephone follow-up.…”

“…ED patients 18 years of age or older were eligible to participate in the survey A total of 26,991 ED DTC adult patients discharged between January and March 2018 from the 16 hospitals were sampled and randomized to nine protocol arms. 13 Eight of these arms were sequential mixed mode protocols (a mode protocol describes the mode[s], sequence, and timing of data collection) involving one or more invitations to a web survey followed by a mailed survey and/or telephone outreach; the reference mode was standard mixed mode (mail with telephone follow-up). The overall response rate was 18.6%, yielding an analytic respondent sample of 4893 adult patients treated in the ED of these 16 hospitals.…”