Is respondents' inattention in online surveys a major issue for research?

Abstract

Participant attentiveness may represent a major concern for all researchers using online self-report survey data, as findings from non-diligent participants add noise and can significantly decrease results reliability. Therefore, attention checks have become a popular method in survey design across social sciences to capture careless or insufficient-effort of respondents, thus increasing quality of samples and the internal validity of the research. The aim of this note is to offer an overview and categorization of the different techniques adopted to flag inattentive respondents and present the potential drawbacks of not considering the issue in social sciences research.

Caught in the Act: Detecting Respondent Deceit and Disinterest in On-Line Surveys. A Case Study Using Facial Expression Analysis

“…Survey structure can maximize attention checks and validity measures. For attention, respondents may be directed to provide a specific answer for some survey items (e.g., “indicate X as your response to this question”); the estimated response time to complete a study can be used as a measure of whether the test was done too quickly; and survey items may query answers given earlier in the process (e.g., age, birthdates), without the means to go back and check (Schroeders et al, 2021; Vecchio et al, 2020). When using Likert-type scales, it may help to lay response choices out in a variety of ways requiring more deliberate attention.…”

“…In the case of attention, social psychologists have called this “careless/insufficient effort” (C/IE) (Curran, 2016; Schroeders et al, 2021; Vecchio et al, 2020). Curran and others suggest that up to 30% of data generated from on-line surveys may be C/IE, with a modal rate near 8–12% (Curran, 2016).…”

Choice Experiment Performed on the Fresh Black Slavonian Pig’s Meat: A Preliminary Study

“…This was a sufficient number of respondents for the pilot study to obtain the valid estimates for the stated preferences (Mariel et al, 2021). In the post-hock analysis according to Vecchio et al (2020), the individuals who had a survey completion and response time lower than five minutes were removed from the sample due to their inattention during the online survey, signifying that the respondents have finished the survey too fast or have answered the questions randomly. Prior to the study's commencement, the respondents were asked to participate therein voluntarily and were screened by three questions, certifying that they were 18 years old, consumed pork, and were (at least partially) responsible for the purchase of foodstuffs in their households.…”

Public perception of ecosystem and social services produced by Sardinia extensive dairy sheep farming systems

“…A structured questionnaire was created in Google Forms in the Italian language and disseminated across social networks. Online surveys are increasingly used to collect data for research purposes, as they are cheap, fast and efficient in collecting valid data, removing geographic limitations and costs (Wright 2005;Vecchio et al 2020). Recruitment was based on non-random criteria and influenced by the use of social networks, creating a potential selection/sampling bias, as respondents' beliefs, interests and strength of feelings about a topic can influence their willingness to participate in a survey (McAleese et al 2016;Burruss and Johnson 2021).…”