Agree or Disagree: Does It Matter Which Comes First? An Examination of Scale Direction Effects in a Multi-device Online Survey

Abstract

Previous research shows that the direction of rating scales can influence participants’ response behavior. Studies also suggest that the device used to complete online surveys might affect the susceptibility to these effects due to the different question layouts (e.g., horizontal grids vs. vertical individual questions). This article contributes to previous research by examining scale direction effects in an online multi-device survey conducted with panelists in Spain. In this experiment, respondents were randomly assigned to two groups where the scale direction was manipulated (incremental vs. decremental). Respondents completed the questionnaire using the device of their choosing (57.8% used PCs; 36.5% used smartphones; and 5.7% used tablets). The results show that scale direction influenced response distributions but did not significantly affect data quality. In addition, our findings indicate that scale direction effects were comparable across devices. Findings are discussed and implications are highlighted.

Polling on police use of body cameras: The effect of response option order on distribution and concurrent validity

“…This is consistent with the literature on response option order effects: options near the beginning of the scale, particularly the first response option respondents perceive as acceptable, are more likely to be chosen (Garbarski, Schaeffer, and Dykema 2019;Krosnick 1991;Yan and Keusch 2015;Yan, Keusch, and He 2018). We also confirmed that the difference in the distribution by response option order is largely similar across device type (desktop/laptop vs. smartphone/tablet, not shown) (Krebs and Höhne 2021;Leon, Aizpurua, and van der Valk, Forthcoming).…”