Drop Downs and Scroll Mice: The Effect of Response Option Format and Input Mechanism Employed on Data Quality in Web Surveys

Abstract

Online researchers face whether to use radio buttons or drop downs when presenting respondents with “select one answer from many” questions. However, empirical evidence regarding response effects does not provide direction for favoring one. Using data collected in a New Zealand general population Web survey of 2,400 people, this study contributes to the decision process by investigating format response effects at multiple levels and exploring the potential for input mechanisms to interfere with drop-down answer selection. Format choice did not significantly affect survey completions, number of nonsubstantial answers, or time to completion. However, drop downs led to higher item nonresponse and longer response times. Furthermore, the 76% of respondents using scroll mice to complete the survey were prone to accidentally changing an answer if presented with drop-down questions. This increased average response times and skewed distribution of responses to the drop-down treatment questions toward the bottom of the response list. Implications of these findings for Web-based data collection are discussed.

Working towards Usable Forms on the Worldwide Web: Optimizing Multiple Selection Interface Elements

“…In an empirical study, Healey [16] found that on the single-question level, radio buttons were faster to choose from than drop-down menus, but the use of drop-down menus instead of radio buttons did not affect the overall time to fill in the whole questionnaire. Hogg and Masztal [17] could not find any differences in the time needed to select answers between radio buttons and drop-down menus.…”

Simple but Crucial User Interfaces in the World Wide Web: Introducing 20 Guidelines for Usable Web Form Design

“…Heerwegh and Loosveldt (2002) found that people needed significantly more time to select options from drop-down menus than from radio buttons, but these findings could not be replicated in a second study. Concerning the drop-out rate, no differences between radio buttons and drop-down menus could be found (Healey, 2007;Heerwegh & Loosveldt, 2002;Hogg & Masztal, 2001). According to Miller and Jarret (2001), radio buttons should be used when two to four options are available; with more than four options they recommend using drop-down menus.…”

“…With the help of the Keystroke-Level Model (Card et al, 1980), it can be theoretically calculated that interaction with a drop-down menu takes longer than interaction with radio buttons, mainly because of the additional click needed to open the drop-down menu. In an empirical study, Healey (2007) found that on the single-question level, radio buttons were faster to choose from than drop-down menus, but the use of dropdown menus instead of radio buttons did not affect the overall time to fill in the whole questionnaire. Hogg and Masztal (2001) could not find any differences in the time needed to select answers between radio buttons and drop-down menus.…”

“…Guideline 9: Use checkboxes instead of list boxes for multiple selection items (see BargasAvila et al, 2009). Guideline 10: For up to four options, use radio buttons; when more than four options are required, use a drop-down menu to save screen real estate (see Healey, 2007;Heerwegh and Loosveldt, 2002;Miller & Jarret, 2001). Guideline 11: Order options in an intuitive sequence (e.g., weekdays in the sequence Monday, Tuesday, etc.).…”

KLM Form Analyzer: Automated Evaluation of Web Form Filling Tasks Using Human Performance Models

“…This is due to an additional point and click needed to open the dropdown menu. However, in one study the latter hypothesis was confirmed [7] and in another it was rejected [8].…”