Impact of question topics and filter question formats on web survey breakoffs

Abstract

Web surveys have become increasingly popular over the last decade, but they tend to suffer from breakoffs, which take place when respondents start the survey but do not complete it. Many studies have investigated the factors impacting breakoffs, but they often ignored the breakoff timing and gave scant attention to two factors: question topics and filter question formats (grouped vs. interleafed as defined by whether filter questions are presented upfront or not). Using survival analysis, this study first identifies when breakoffs are more likely to happen and what are the time-varying predictors of breakoffs. Then, by using a web survey that experimentally manipulates the filter question format and randomly orders the question topic, this study investigates the effect of question topics and filter question formats on the breakoff event and its timing. We find that most breakoffs tend to happen at the beginning of the questionnaire and at the place where a new question topic is introduced. While item nonresponse is associated with more breakoffs, it is surprising to see that open-ended and long questions are associated with a lower breakoff risk. Additionally, we discover that grouping the filter questions leads to fewer breakoffs at the beginning compared to the interleafed counterpart, but the breakoff risk in the grouped format catches up quickly when respondents realise their previous answers will trigger more questions. This study also shows that questions about insurance have more breakoffs while questions on demographics and income have fewer breakoffs despite their sensitivity level.

Diary studies in research: More than a research method

“…Although Z. Chen et al (2022b) suggest that break-off in question sequence was higher when the responded realises that their answer would trigger more questions, this was not observed in this study. The design however employed a logic that introduced and explored themes based on response to carefully worded closed questions, and simulated a semi-structured interview (Saunders et al, 2019).…”

“…It is therefore recommended that once the commitment of the intended diarist has been confirmed, detailed training sessions should be conducted so that the participant understand both the protocol and entry requirements (Reis & Gable, 2000). Additionally, it is worthwhile that researchers design studies in such a way that questions, when used, can be completed quickly (Bolger et al, 2003;Z. Chen et al, 2022b).…”

Predicting Web Survey Breakoffs Using Machine Learning Models

“…It can explain whether breakoff will happen and if so its timing (Singer & Willett, 2003). We code the timing as the number of questions seen by respondents up to the point where they broke off or completed the survey, which is an approach commonly used in the prior literature (Chen et al, 2022; Mittereder & West, 2021; Peytchev, 2009).…”

“…The Cox survival model is commonly used in the study of survey breakoffs (Chen et al, 2022; Hochheimer et al, 2016; Mittereder & West, 2021; Peytchev, 2009). This model estimates the breakoff hazard which is defined as the probability of a person breaking off at a specific question given that this person has not experienced this event before (Mills, 2011).…”

“…The Cox survival model is widely used when studying survey breakoffs (Chen et al, 2022; Hochheimer et al, 2016; Mittereder, 2019; Peytchev, 2009) as not breaking off implies “surviving” the response process. Previous research has shown that this traditional model can achieve a relatively satisfactory prediction accuracy (e.g., 78% for Mittereder, 2019).…”