Attention Checks in Decisions From Experience: The Case of Checking Experiments

Abstract

Can online participants be compared to laboratory subjects. In the five studies (Ntotal = 1519) reported in this article, we comprehensively compared the behavior of online attentive and inattentive participants; i.e., those who passed or failed a simple attention check. The findings show that in a decisions from experience paradigm (i.e., a multi-trial repeated choice task), even one simple attention check is sufficient to differentiate between attentive and inattentive participants. The validity of this separation is further evidenced in the significant difference in the reported conscientiousness measure.The results fully replicated three previously run lab studies for the attentive participants, but not for the inattentive participants. This highlights the importance of using attention checks to avoid spurious conclusions. There are three explanations for the behavioral differences between the attentive and inattentive participants. Attentive participants exhibited a pre-disposition towards checking. They also had a greater likelihood of noticing distinct outcomes. Finally, the inattentive participants tended to behave more randomly. When checking was beneficial, these three effects pointed in the same direction, leading to a major difference between participants. When checking was detrimental, these effects cancelled each other out, resulting in similar checking rates and indistinguishable behavior.

Contradictory deviations from maximization: Environment-specific biases, or reflections of basic properties of human learning?

“…Each task was faced by 20 or more attentive participants.Each participant could participate in one session per day, and each session focused on only one task (100 trials). The attention check was conducted after the participants completed the 100 trials of the experiment (see Roth & Yakobi, 2022). It required a choice one among five alternatives (A, B, …, E) with their respective outcomes presented before they made their choice, as shown in Figure A1…”

“…14 The experiment used the standard method described above (and clarified in Appendix A). Appendix A presents the results (of at least 20 participants who passed the attention test in each task, Roth & Yakobi, 2022, the total number of participants was 2,702). The two right-hand columns in Table 3 present two measures of the models' prediction accuracy used above: the normalized bias, and the LogLoss scores.…”

“…The code presented below uses the Python package Decision from Experience Behavior Modeling (Yakobi & Roth, 2022).…”