Predicting Web Survey Breakoffs Using Machine Learning Models

Abstract

Web surveys are becoming increasingly popular but tend to have more breakoffs compared to the interviewer-administered surveys. Survey breakoffs occur when respondents quit the survey partway through. The Cox survival model is commonly used to understand patterns of breakoffs. Nevertheless, there is a trend to using more data-driven models when the purpose is prediction, such as classification machine learning models. It is unclear in the breakoff literature what are the best statistical models for predicting question-level breakoffs. Additionally, there is no consensus about the treatment of time-varying question-level predictors, such as question response time and question word count. While some researchers use the current values, others aggregate the value from the beginning of the survey. This study develops and compares both survival models and classification models along with different treatments of time-varying variables. Based on the level of agreement between the predicted and actual breakoff, we find that the Cox model and gradient boosting outperform other survival models and classification models respectively. We also find that using the values of time-varying predictors concurrent to the breakoff status is more predictive of breakoff, compared to aggregating their values from the beginning of the survey, implying that respondents’ breakoff behaviour is more driven by the current response burden.

Breakoffs in an hour-long, online survey

“…Existing literature on breakoff rates rarely examine a survey of such length (see Table 1), and it is unknown whether breakoff rates beyond 30 minutes remain steady, accelerate or decline, and whether this is dependent on the context of the surveys such as the device type used, incentives used or time of day of starting the survey. It is therefore difficult to assess the impact of breakoffs on data quality based on existing findings which focus on very short survey lengths (Chen, Cernat, and Shlomo 2023;Mittereder and West 2022).…”