Motion instructions in surveys: Compliance, acceleration, and response quality

Abstract

The increased use of smartphones in web survey responding did not only raise new research questions but also fostered new ways to research survey completion behavior. Smartphones have many built-in sensors, such as accelerometers that measure acceleration (i.e., the rate of change of velocity of an object over time). Sensor data establish new research opportunities by providing information about physical completion conditions that, for instance, can affect response quality. In this study, we explore three research questions: (1) To what extent do respondents accept to comply with motion instructions? (2) What variables affect the acceleration of smartphones? (3) Do different motion levels affect response quality? We conducted a smartphone web survey experiment using the Netquest opt-in panel in Spain and asked respondents to stand at a fix point or walk around while answering five single questions. The results reveal high compliance with motion instructions, with compliance being higher in the standing than in the walking condition. We also discovered that several variables, such as the presence of third parties, increase the acceleration of smartphones. However, the quality of responses to the five single questions did not differ significantly between the motion conditions, a finding that is in line with previous research. Our findings provide new insights into how compliance changes with motion tasks and suggest that the collection of acceleration data is a feasible and fruitful way to explore survey completion behavior. The findings also indicate that refined research on the connection between motion levels and response quality is necessary.

Completion Conditions and Response Behavior in Smartphone Surveys: A Prediction Approach Using Acceleration Data

“…In terms of the outcome variable, we focus on a binary variable that condenses the four conditions of the lab experiment into the categories moving (i.e., walking and stair climbing) and nonmoving (i.e., sitting and standing). In line with previous research, we expect that these physiological states are most informative when studying completion conditions in smartphone surveys (see Höhne, Revilla, et al, 2020;Höhne & Schlosser, 2019).…”

“…Beyond the use of paradata to detect multitasking, Höhne and Schlosser (2019) and Toepoel and Lugtig (2015) suggest to passively collect sensor data in smartphone surveys to draw conclusions about respondents' completion conditions. Accelerometers measure the rate of change of velocity of an object over time, allowing researchers to unobtrusively record physiological states, such as movements (see Elhoushi et al, 2017;Harari et al, 2016;He et al, 2016;Höhne, Revilla, et al, 2020;Höhne & Schlosser, 2019;Toepoel & Lugtig, 2015). If a person moves or walks, they are creating acceleration (He et al, 2016), which is detected by the smartphone that is commonly worn on the body (e.g., in the pocket).…”

“…If a person moves or walks, they are creating acceleration (He et al, 2016), which is detected by the smartphone that is commonly worn on the body (e.g., in the pocket). This situation can be applied to respondents that have the smartphone in their hands and complete a web survey, referring to a "respondent-device" link (Höhne, Revilla, et al, 2020;Höhne & Schlosser, 2019). To put it differently, respondents' motions are detectable by the acceleration sensor of smartphones, allowing researchers to classify smartphone respondents on the basis of their motion levels (i.e., acceleration).…”

“…In contrast to paradata, which can be used to study on-device multitasking, the usage of acceleration data provides an avenue to learn about the (situational) conditions in which respondents complete web surveys. For instance, Höhne, Revilla, et al (2020) show that completing surveys outdoors, moving around during survey completion, and the presence of third parties during survey completion significantly increase the acceleration of smartphones. Thus, acceleration data may be used to study distractions that go beyond respondent-initiated multitasking activities, without relying on self-reports.…”

Augmenting Surveys With Data From Sensors and Apps: Opportunities and Challenges

“…No one solution exists to "fix" the missing data problem. Some principled methods help mitigate missing GPS location data (e.g., Barnett & Onnela, 2020) and accelerometer data (e.g., Höhne et al, 2019;Lee & Gill, 2018). However, how to deal with missing sensor data as a whole, beyond the individual sensors, is still an open issue.…”