Online versus offline: The Web as a medium for response time data collection

Abstract

The Internet provides a convenient environment for data collection in psychology. Modern Web programming languages, such as JavaScript or Flash (ActionScript), facilitate complex experiments without the necessity of experimenter presence. Yet there is always a question of how much noise is added due to the differences between the setups used by participants and whether it is compensated for by increased ecological validity and larger sample sizes. This is especially a problem for experiments that measure response times (RTs), because they are more sensitive (and hence more susceptible to noise) than, for example, choices per se. We used a simple visual search task with different set sizes to compare laboratory performance with Web performance. The results suggest that although the locations (means) of RT distributions are different, other distribution parameters are not. Furthermore, the effect of experiment setting does not depend on set size, suggesting that task difficulty is not important in the choice of a data collection method. We also collected an additional online sample to investigate the effects of hardware and software diversity on the accuracy of RT data. We found that the high diversity of browsers, operating systems, and CPU performance may have a detrimental effect, though it can partly be compensated for by increased sample sizes and trial numbers. In sum, the findings show that Web-based experiments are an acceptable source of RT data, comparable to a common keyboard-based setup in the laboratory.

Gone in sixty (milli)seconds: Disentangling the effects of location context on attention bias

“…Additionally, it is worth noting that RTs during the EST for participants in the neutral condition did not differ as a function of location. This finding is at odds with several studies showing significantly slower overall RTs remotely (Chetverikov & Upravitelev, ; Reimers & Stewart, ), but consistent with a recent study showing no difference in overall RTs between the laboratory and remotely (Miller et al ., ). The mixed findings in the literature makes it difficult to determine why our findings were not internally consistent, although in the context of the current study the emotion provocation may have played a role in the slower overall RTs found remotely.…”

“…All study procedures were approved by the Institutional Review Board of the University of Texas at Austin. Based on simulation experiments conducted by Chetverikov and Upravitelev () we determined that approximately 80 participants per group would provide sufficient power to detect a small (10 ms) difference in RTs.…”

“…Assessing within individual variation in attention bias as a function of assessment context (i.e., laboratory vs. home) and the factors contributing to that variation (e.g., mood state) have important implications for both ABM treatment and efforts to assess attentional bias in more naturalistic settings. Several studies have examined within subject differences across assessment contexts on perceptual performance RT tasks and have generally found equivalent task effects, however while earlier research found that RTs were slower overall across tasks when they were completed remotely (Chetverikov & Upravitelev, ; Reimers & Stewart, ), a more recent study with a larger sample found that there was no difference in overall RTs (Miller, Schmidt, Kirschbaum & Enge, ). Additionally, there is a substantial body of evidence that cognitive (e.g., Stroop) and perceptual (e.g., visual search) RT task effects are consistently found when completed remotely (Barnhoorn, Haasnoot, Bocanegra & van Steenbergen, ; Crump, McDonnell & Gureckis, ; Germine et al ., ; Hilbig, ).…”

Promises and pitfalls of Web-based experimentation in the advance of replicable psychological science: A reply to Plant (2015)

“…Although methodological challenges will need to be addressed in future optimizations, an increasing number of studies indicate that the timing characteristics of browser-based experiments are acceptable enough to warrant the replication of default laboratory experiments (Barnhoorn, Haasnoot, Bocanegra, & van Steenbergen, 2015; Chetverikov & Upravitelev, 2015; de Leeuw, 2015; de Leeuw & Motz, 2015; Garaizar, Vadillo, & López-de-Ipiña, 2014; Keller, Gunasekharan, Mayo, & Corley, 2009; Neath, Earle, Hallett, & Surprenant, 2011; Reimers & Stewart, 2015; Schubert, Murteira, Collins, & Lopes, 2013; Slote & Strand, 2015; von Bastian, Locher, & Ruflin, 2013). In particular, the introduction of HTML5 presentation techniques has enabled frame-wise stimulus presentation for browser-based experiments, a method we have included in the QRTEngine (Barnhoorn et al., 2015), which allows millisecond accuracy to be approached (Garaizar et al., 2014).…”

Web-based Platform for Psychology Research

“…The psychological society tends not to trust the precision of tests administered online because of the potential confounding factors such as the technical properties of the personal computers or monitors of the respondents. However, the quantitative estimation of the noise added due to the differences in setup shows that at least for some type of tasks web-based experiments can be an acceptable source of data [26]. Thus, the next logical step for the computerized cognitive tests is to spread across the internet the same way it has happened to the questionnaires.…”