Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies

Abstract

Increasingly, experimental research is being conducted on the Internet in addition to the laboratory. Online experiments are more convenient for subjects and researchers, but we know little about how the choice of study location affects data quality. To investigate whether respondent behavior differs across study location, we randomly assign subjects to participate in a study in a laboratory or in an online setting. Contrary to our expectations, we find few differences between participants in terms of the level of attention and socially desirable responding. However, we find significant differences in two areas: the degree of selfreported distractions while completing the questionnaire and the tendency to consult outside sources for answers to political knowledge questions. We conclude that when the greater convenience (and higher response rates) of online experiments outweighs these disadvantages, Internet administration of randomized experiments represent an alternative to laboratory administration.

Building, Hosting and Recruiting: A Brief Introduction to Running Behavioral Experiments Online

“…In addition to timing, there could be concerns that participants might be less committed when they sit at home and are not directly observed by the experimenter. Several studies have shown that decreased attention to the task is not necessarily found [63,64] and data quality is comparable to lab-based studies [65][66][67][68][69][70][71][72][73]. For example, in a recent study, participants completed several attention checks (in between outcome measures) and there was no difference between lab participants and online participants in any of the measures.…”

Go Fix It: Comedy as an Agent of Political Activation*

“…The study is limited by its sample, obtained through mTurk. Although a wealth of research now shows that such samples are comparable to others, especially for experimental research (Berinsky, Huber, and Lenz, ; Clifford and Jerit, ; Levay, Freese, and Druckman, ; Necka et al., ), the sample is not representative, particularly on education, partisanship, and race, and there is some evidence that MTurk workers are more attentive (Hauser and Schwarz, ) and more aware of experiments (Chandler, Mueller, and Paolacci, ) than other experimental participants, and therefore we cannot be confident that other samples would respond similarly to our stimuli. It is further limited in that we consider only a single (edited) episode of LWT , so we cannot be sure that these effects generalize to calls to action more generally, or to other political comedy programs.…”

Positive Parenting and Parental Conflict: Contributions to Resilient Coparenting During Divorce

“…Participants were compensated using the Qualtrics Panels workforce system, which was the equivalent of $4 for the presurvey and $2 for the postsurvey. These types of online marketplace services are growing in their use by extension professionals (Dworkin, Brar, Hessel, & Rudi, 2016a) and social science researchers, especially to recruit for hard-to-reach samples (Smith, Sabat, Martinez, Weaver, & Xu, 2015), and few differences have been found between traditionally recruited samples and online samples (Clifford & Jerit, 2014;Dworkin, Hessel, Gliske, & Rudi, 2016b). Participants for the original study were invited to complete a postsurvey 3 months later; however, only data from the initial survey was used for the present study.…”