A Dynamic Survival Modeling Approach to the Prediction of Web Survey Breakoff

Abstract

Respondents who break off from a web survey prior to completing it are a prevalent problem in data collection. To prevent breakoff bias, it is crucial to keep as many diverse respondents in a web survey as possible. As a first step of preventing breakoffs, this study aims to understand breakoff and the associated response behavior. We analyze data from an annual online survey using dynamic survival models and ROC analyses. We find that breakoff risks between respondents using mobile devices versus PCs do not differ at the beginning of the questionnaire, but the risk for mobile device users increases as the survey progresses. Very fast respondents as well as respondents with changing response times both have a higher risk of quitting the questionnaire, compared to respondents with slower and steady response times. We conclude with a discussion of the implications of these findings for future practice and research in web survey methodology.

Impact of question topics and filter question formats on web survey breakoffs

“…As a result, a framework has been developed to summarise different factors. According to Peytchev (2009) and Mittereder and West (2021), these factors can be grouped into four categories: (1) page/ question characteristics, (2) survey design, (3) respondent factors and (4) paradata.…”

“…Paradata, the final category in the framework, refer to the information collected during the response process (Kreuter, 2013). This type of data is believed to reflect the change in the response burden and respondents' motivation throughout the questionnaire (Mittereder & West, 2021), thereby being useful for predicting the imminent breakoff. Some paradata that have been associated with breakoffs are the proportion of questions that are not answered (Mittereder & West, 2021) and using mobile devices to answer the survey (Wenz, 2017).…”

“…As an alternative to not answering the sensitive question, respondents might terminate their survey participation. However, among those papers that investigated the relationship between question topics and breakoffs (see McGonagle, 2013;Mittereder & West, 2021), the order of the topics was not randomised. For example, McGonagle (2013) analysed a telephone survey about the U.S. families' economic status, but the topic about respondents' housing was always followed by their employment history, income and so on.…”

Classification ensembles for multivariate functional data with application to mouse movements in web surveys

“…In CAI and web survey modes, for example, response times have been one of the most popular paradata types considered for predicting break-offs in web surveys and designing interventions (Mittereder and West, 2021), studying the relationship of response times and measurement errors (Heerwegh, 2003) or detecting sources of difficulty in the survey (Conrad et al, 2007;Yan and Tourangeau, 2008). However, response times are not always reliable descriptors of the entire survey process (e.g., a long response time may not necessarily be due to the survey question but to other nonsurvey-related tasks such as checking or responding to emails, see Horwitz et al, 2017;Fernández-Fontelo et al, 2021).…”

Evaluation of User Reactions and Verification of the Authenticity of the User’s Identity during a Long Web Survey

“…Research in works [8][9][10][11] is dedicated to web questionnaire design and emphasize that the user reaction time should be noted. Works [12][13][14] consider reaction time to predict that the respondent will abandon a web survey prior to completing it. It is logical to assume that there is a correlation between the time it takes to read a question and the reaction time; besides, a direct comparison of respondents' reaction times is problematic, because each person has an individual speed [15].…”