Factors Affecting Completion Times: A Comparative Analysis of Smartphone and PC Web Surveys

Abstract

This article compares the factors affecting completion times (CTs) to web survey questions when they are answered using two different devices: personal computers (PCs) and smartphones. Several studies have reported longer CTs when respondents use smartphones than PCs. This is a concern to survey researchers because longer CTs may increase respondent burden and the risk of breakoff. However, few studies have analyzed the specific reasons for the time difference. In this analysis, we analyzed timing data from 836 respondents who completed the same web survey twice, once using a smartphone and once using PC, as part of a randomized crossover experiment in the Longitudinal Internet Studies for the Social Sciences panel. The survey contained a mix of questions (single choice, numeric entry, and text entry) that were displayed on separate pages. We included both page-level and respondent-level factors that may have contributed to the time difference between devices in cross-classified multilevel models. We found that respondents took about 1.4 times longer when using smartphones than PCs. This difference was larger when a page had more than one question or required text entry. The difference was also larger among respondents who had relatively low levels of familiarity and experience using smartphones. Respondent multitasking was associated with slower CTs, regardless of the device used. Practical implications and avenues for future research are discussed.

Is There More Than the Answer to the Question? Device Use and Completion Time as Indicators for Selectivity Bias and Response Convenience in Online Surveys

“…Nearly all previous studies comparing response times across desktop and mobile device modes have found that web surveys take longer to complete on mobile devices than on desktop devices (Andreadis 2015;Antoun and Cernat 2019;Couper and Peterson 2017;Schlosser and Mays 2018). Moreover, meta analyses of 21 studies (Gummer and Roßmann 2015) and 26 studies (Couper and Peterson 2017) corroborate these results.…”

“…For these purposes, paradata offer objective indicators of response behaviour, data quality, and usability of the survey (e.g. Antoun and Cernat 2019;Brockhaus et al 2017;Couper and Peterson 2017;Mayerl and Giehl 2018;McClain et al 2019;Roßmann and Gummer 2016;Sendelbah et al 2016). Moreover, paradata can be used to reflect and interpret responses from a content perspective (Yan and Olson 2013) or to classify respondents' personalities based on indicators of response behaviour (Stieger and Reips 2010).…”

“…Response times are often used as an indicator to compare survey burden and survey convenience across modes, which are central to predicting current and future dropouts in a panel survey (Antoun and Cernat 2019). The longer a survey takes, the higher is the survey burden.…”

Breakoffs in an hour long online survey

“…This problem can be aggravated in the case of long surveys as the respondent burden increases. Antoun & Cernat (2020), using LISS data, found that smartphone respondents took on average 40% longer than respondents using personal computers (PCs) to complete the survey, noting also that more respondents engaged in multitasking when using smartphones.…”

Testing a Planned Missing Design to Reduce Respondent Burden in Web and SMS Administrations of the CAHPS Clinician and Group Survey (CG-CAHPS)

“…The planned missing design is used to manipulate respondent burden as the shortened questionnaire modules are hypothesized to relieve respondent burden relative to the fulllength questionnaire module. Following previous theory and research, respondent burden is operationalized as survey duration, and reflected in breakoff rates and, accordingly, completion rates (Antoun and Cernat 2020;Bradburn 1978;Mavletova and Couper 2015;Peytchev 2009;Sharp and Frankel 1983;Steinbrecher et al 2015). Breakoff rates are computed as the percentage of eligible respondents who started the survey but did not complete it.…”