Designing Scalar Questions for Web Surveys

Abstract

This paper explores how the visual design of scalar questions influences responses in web surveys. We present the results of five experiments embedded in two web surveys of university students. We find that consistently presenting the positive end of the scale first did not impact responses but increases response times. Displaying the categories in multiple columns influence how respondents process the scale and increase response times. Separating the midpoint, ``don't know'' option, or endpoints spatially does not impact responses when the visual and conceptual midpoint align. Removing the graphical layout of the scale influences responses when lower numbers indicate more positive categories and increases response time. Finally, response times are longer for polar point scales with numeric labels, but there are no differences in responses. Overall, our results suggest that the visual design of response scales impacts measurement, but that some manipulations produce larger and more significant differences than others.

Improving web survey efficiency: the impact of an extra reminder and reminder content on web survey response

“…The advantages of web surveys have been extensively documented in the academic literature. 1 They tend to reduce the cost of questionnaire distribution and administration and eliminate the influence of an interviewer (Callegaro et al, 2015;Couper, 2000;Tourangeau, Couper, & Conrad, 2004), while respondents are able to control how and when they complete the survey (Callegaro et al, 2015;Christian, Parsons, & Dillman, 2009) and no educational time is lost by the completion of questionnaires or evaluations during lectures. In addition, web surveys offer the advantages of obtaining large samples in a relatively easy way (Couper, 2000;Malhotra, 2008) as well as increased response accuracy because respondents enter their own information directly (Durrant & Dorius, 2007).…”

Visualization of Questionnaire in Online Surveys: How Different Design Features Impact the Data Quality

“…Поэтому для повышения точности содержательных данных авторы рекомендовали давать более строгое визуаль-ное разграничение между содержательными и несодержательными вариантами ответов. Л. М. Кристиан, Н. Парсонс и Д. Диллман в качестве экспериментального фак-тора использовали другой способ визуального выделения несодержательных аль-тернатив -они увеличили интервал между последним содержательным и первым несодержательным вариантами [Christian, Parsons, Dillman, 2009]. Результаты эксперимента подтвердили выводы Туранжо, Купера и Конрада об увеличении доли несодержательных ответов при визуальном их выделении, однако вступили в противоречие с выводами о влиянии визуализации несодержательных вариан-тов ответа на структуру содержательных ответов: значимо более высокие средние оценки по шкале были получены в случае визуального выделения несодержа-тельной альтернативы.…”

Learning and Satisficing: An Analysis of Sequence Effects in Health Valuation

“…(5–7) Experimental design, such as the layout of questions, number of attributes, and number of tasks, can influence ordering effects and response time. (8–10) A key example in survey research is the primacy effect or the tendency for respondents to choose the first reasonable answer to a survey question (e.g., first response option in a list of potential answers). (6, 11) This weak form of satisficing leads to non-random response; expedites response with minimum effort; reduces response quality and time; and is commonly cited by experimenters in order to justify randomization and reduction of the number of attributes, scenarios, and tasks.…”