Completion Conditions and Response Behavior in Smartphone Surveys: A Prediction Approach Using Acceleration Data

Abstract

This study utilizes acceleration data from smartphone sensors to predict motion conditions of smartphone respondents. Specifically, we predict whether respondents are moving or nonmoving on a survey page level to learn about distractions and the situational conditions under which respondents complete smartphone surveys. The predicted motion conditions allow us to (1) estimate the proportion of smartphone respondents who are moving during survey completion and (2) compare the response behavior of moving and nonmoving respondents. Our analytical strategy consists of two steps. First, we use data from a lab experiment that systematically varied motion conditions of smartphone respondents and train a prediction model that is able to accurately infer respondents’ motion conditions based on acceleration data. Second, we use the trained model to predict motion conditions of respondents in two cross-sectional surveys in order to compare response behavior of respondents with different motion conditions in a field setting. Our results indicate that active movement during survey completion is a relatively rare phenomenon, as only about 3%–4% of respondents were predicted as moving in both cross-sectional surveys. When comparing respondents based on their predicted motion conditions, we observe longer completion times of moving respondents. However, we observe little differences when comparing moving and nonmoving respondents with respect to indicators of superficial responding, indicating that moving during survey completion does not pose a severe threat to data quality.

Is there a growing use of mobile devices in web surveys? Evidence from 128 web surveys in Germany

“…For instance, GPS data provide information about respondents' geolocation and, thus, they can be used to infer the environmental setting (Kelly et al 2013;Struminskaya et al 2020). Similarly, acceleration data can help to provide information about the different motion conditions of smartphone respondents, such as standing or walking, during survey completion (Kern et al 2020). However, utilizing these potentials of mobile devices will require researchers to overcome device effects on consent behavior that have been reported by prior research (Wenz et al 2019).…”

Using Smartphone Sensor Paradata and Personalized Machine Learning Models to Infer Participants’ Well-being: Ecological Momentary Assessment

“…Data from acceleration and orientation sensors have also allowed researchers to infer participants' emotions [40,41]. Furthermore, Kern et al [42] showed that the accelerometer data recorded alongside the survey provided information about survey completion conditions (eg, whether a participant moved while taking the survey).…”

“…Consequently, a very poor-performing model might simply originate from a participant's irregular behavior when filling out the questionnaire. However, contrary to the same issues with self-report data, combining sensors with pretrained classification algorithms might be fruitful for determining participants' activity levels for each measurement and informing the models beyond the aggregated features [25,42].…”