I’m paid biweekly, just not by leprechauns: Evaluating valid-but-incorrect response rates to attention check items

Abstract

Participant carelessness is a source of invalidity in psychological data (Huang, Liu, & Bowling, 2015), and many methods have been created to screen for this carelessness (Curran, 2016; Johnson, 2005). These include items that researchers presume thoughtful individuals will answer in a given way (e.g., disagreement with "I am paid biweekly by leprechauns", Meade & Craig, 2012). This paper reports on two samples in which individuals spoke aloud a series of these questions, and found that a) individuals do occasionally report valid justifications for presumed invalid responses, b) there is relatively high variance in this behavior over different items, and c) items developed for this specific purpose tend to work better than those drawn from other sources or created ad-hoc.

Noncompliant responding: Comparing exclusion criteria in MTurk personality research to improve data quality

“…The statistical approach can also compensate for legitimate concerns about the use of intuitive-but overly stringent-zero-tolerance exclusion criteria to attention check questions (Curran & Hauser, 2018). Our findings show that when the recommended lenient approach was used to flag responses on the instructed response questions (e.g., Curran, 2016), the number of flagged responses dropped markedly.…”

“…Our results are also informative for preregistration procedures. As noted in the introduction, depending on the source and the type of check question, lenient or a zero-tolerance thresholds are suggested to flag responses on attention check questions (Curran, 2016;Curran & Hauser, 2018;Kim et al, 2018). This may have led researchers to often apply their own procedures to exclude responses (or similarly, to not exclude any responses).…”

“…However, these check questions have advantages and drawbacks. Although infrequency items easily blend in the survey, they may create some confusion because some people interpret infrequency questions figuratively (Curran & Hauser, 2018). For instance, these authors found some people agreed with the question 'I get paid by leprechauns biweekly' if they got paid bi-weekly.…”

“…An advantage of instructed response items is that they are unlikely to cause confusion (Kim et al, 2018). However, Curran and Hauser (2018) found that up to 11% of respondents gave a wrong response to these questions. Consequently, the zero-tolerance approach also seems inappropriate when flagging responses to these questions.…”

Careless Response Processes are Heterogeneous: Comment on Goldammer et al. (2020)

“…Last, a fourth approach might be to induce content non-responsive responses by using items with nonsensical or blank content (cf. Maul, 2017; but see also Rhemtulla et al, 2017 andCurran &Hauser, 2019).…”