Promises and pitfalls of Web-based experimentation in the advance of replicable psychological science: A reply to Plant (2015)

Abstract

In a recent letter, Plant (2015) reminded us that proper calibration of our laboratory experiments is important for the progress of psychological science. Therefore, carefully controlled laboratory studies are argued to be preferred over Web-based experimentation, in which timing is usually more imprecise. Here we argue that there are many situations in which the timing of Web-based experimentation is acceptable and that online experimentation provides a very useful and promising complementary toolbox to available lab-based approaches. We discuss examples in which stimulus calibration or calibration against response criteria is necessary and situations in which this is not critical. We also discuss how online labor markets, such as Amazon’s Mechanical Turk, allow researchers to acquire data in more diverse populations and to test theories along more psychological dimensions. Recent methodological advances that have produced more accurate browser-based stimulus presentation are also discussed. In our view, online experimentation is one of the most promising avenues to advance replicable psychological science in the near future.

Mental chronometry in the pocket? Timing accuracy of web applications on touchscreen and keyboard devices

“…RT overestimation varied across devices, similar to what was found in previous research (Neath et al, 2011;Reimers & Stewart, 2015). The range of mean RT overestimations was similar to or smaller than the distributions assumed in various simulation studies with between-group designs (Brand & Bradley, 2012;Reimers & Stewart, 2015;Vadillo & Garaizar, 2016). The iOS device had the lowest mean RT overestimations, whereas MacOS had the highest.…”

“…With regard to RT measurement, research has indicated a noisy overestimation of RTs, with the mean and variance of overestimations varying across devices and browsers (Neath, Earle, Hallett, & Surprenant, 2011;Reimers & Stewart, 2015). In simulation studies, nonsystematic overestimation of RTs has generally been modeled as uniform distributions ranging up to 18 ms (Damian, 2010), 70 ms (Reimers & Stewart, 2015), or 90 ms (Brand & Bradley, 2012;Vadillo & Garaizar, 2016). Such RT overestimation was generally found to have a modest impact on a range of parameter estimation methods and designs, especially when scoring a task by subtracting RTs of a participant between two or more conditions.…”

Multisite Replication in Second Language Acquisition Research: Attention to Form During Listening and Reading Comprehension

“…A second potential issue is that, although the timing specifics of the software programs used to present the passage are not expected to differ meaningfully, the timing specifics of different hardware configurations may have impacted the reaction time output (Stahl, ). Future multisite studies collecting reaction time or online data will need to consider validating the timing of different software and hardware systems (Plant, ), although for paradigms that do not require precise millisecond timing, the benefits of collecting data across larger samples may outweigh disadvantages in timing variability (van Steenbergen & Bocanegra, ). Given these issues, we are not confident in being able to offer a valid interpretation of the reaction time data, but we do provide access to the data and results on our public Open Science Framework analysis page (https://osf.io/nz3su) in the folder for Written Data Analysis under Files.…”

Metronome LKM: An open source virtual keyboard driver to measure experiment software latencies

“…The results of Studies 1 and 2 allow us to conclude that, in principle (and setting aside the potential limitations of hardware; see Plant, 2016, andvan Steenbergen &Bocanegra, 2016, for a discussion of this issue), it is possible to collect user interaction data with standard Web technologies with precision and accuracy below a millisecond. However, these results also reveal a limitation in the collection of reaction times with standard DOM events.…”