The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions.

Abstract

The authors find that experimental studies using online samples (e.g., MTurk) often violate the assumption of random assignment, because participant attrition-quitting a study before completing it and getting paid-is not only prevalent, but also varies systemically across experimental conditions. Using standard social psychology paradigms (e.g., ego-depletion, construal level), they observed attrition rates ranging from 30% to 50% (Study 1). The authors show that failing to attend to attrition rates in online panels has grave consequences. By introducing experimental confounds, unattended attrition misled them to draw mind-boggling yet false conclusions: that recalling a few happy events is considerably more effortful than recalling many happy events, and that imagining applying eyeliner leads to weight loss (Study 2). In addition, attrition rate misled them to draw a logical yet false conclusion: that explaining one's view on gun rights decreases progun sentiment (Study 3). The authors offer a partial remedy (Study 4) and call for minimizing and reporting experimental attrition in studies conducted on the Web. (PsycINFO Database Record

Conducting Online Behavioral Research Using Crowdsourcing Services in Japan

“…First, Zhou and Fishbach (2016) claimed that researchers should pay attention to attrition rate that poses a threat to internal validity of the study. They also recommended that researchers not only implement dropout-reduction strategies, but also explore causes of, increase the visibility of, and report participant attrition.…”

Poor peer work does not boost student confidence

“…Following the suggestion of Zhou and Fishbach (), we examined whether incomplete responses to the experiment materials were unevenly distributed across conditions. Indeed, we noted that 46 individuals dropped out of the study at some time after being assigned to condition, and the percentage of dropouts was higher among those who saw very good work (13.2%) than among those who saw satisfactory or bad work (4.9% and 3.7% respectively), χ 2 (2) = 16.87, p < .001.…”

Self‐punishment promotes reconciliation with third parties by addressing the symbolic implications of wrongdoing

“…Scale reliabilities are given using Cronbach's alpha scores, except for the remorse scale, which has only three items, since scales with few items can bias Cronbach's alpha below the normally accepted standard of .7 (Cortina, 1993). This included a commitment device at the beginning of the survey asking potential participants to affirm that they would answer questions to the best of their ability (Zhou & Fishbach, 2016); a reading check (that ensured participants read the vignette properly; if not they were asked to re-read it and answer this question again, but they were not excluded from the study), and an instructed response item (embedded later in the questionnaire that was used to exclude careless respondents; Meade & Craig, 2012). All measures showed good internal reliability.…”