Timing accuracy of Web experiments: A case study using the WebExp software package

Keller, Frank; Gunasekharan, Subahshini; Mayo, Neil; Corley, Martin

doi:10.3758/brm.41.1.12

Cited by 93 publications

(66 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have been several successful attempts to replicate effects previously observed offline using online samples: Reimers and Maylor (2005) replicated the effects of age differences on task-switching performance measured with RTs; Keller et al (2009) replicated RT findings from a self-paced reading study; Simcox and Fiez (2014) reproduced flanker and lexical decision effects; and Crump et al (2013) replicated Stroop, task-switching, flanker, Simon, visual-cuing, and masked-priming effects. Although these findings show the usefulness of online studies, they do not allow for a direct comparison between offline and online results, because they were obtained in different studies.…”

mentioning

confidence: 67%

“…Schubert et al used their own Flash-based scripting library, ScriptingRT (Schubert, Murteira, Collins, & Lopes, 2013). ScriptingRT was slightly slower and more deviant than most of the offline software, but in absolute terms the SDs were less than 5 ms. Keller et al used repetitive keypress events generated by the operating system when a key was held down to test their Java-based Webexp software (Keller, Gunasekharan, Mayo, & Corley, 2009). They found that although there was a delay at shorter time intervals, it was not big (about 12 ms on 300 ms of repetitive keypresses), and the SDs were less than 5 ms in most of the conditions.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Online versus offline: The Web as a medium for response time data collection

Chetverikov

Upravitelev²

2015

Behav Res

View full text Add to dashboard Cite

The Internet provides a convenient environment for data collection in psychology. Modern Web programming languages, such as JavaScript or Flash (ActionScript), facilitate complex experiments without the necessity of experimenter presence. Yet there is always a question of how much noise is added due to the differences between the setups used by participants and whether it is compensated for by increased ecological validity and larger sample sizes. This is especially a problem for experiments that measure response times (RTs), because they are more sensitive (and hence more susceptible to noise) than, for example, choices per se. We used a simple visual search task with different set sizes to compare laboratory performance with Web performance. The results suggest that although the locations (means) of RT distributions are different, other distribution parameters are not. Furthermore, the effect of experiment setting does not depend on set size, suggesting that task difficulty is not important in the choice of a data collection method. We also collected an additional online sample to investigate the effects of hardware and software diversity on the accuracy of RT data. We found that the high diversity of browsers, operating systems, and CPU performance may have a detrimental effect, though it can partly be compensated for by increased sample sizes and trial numbers. In sum, the findings show that Web-based experiments are an acceptable source of RT data, comparable to a common keyboard-based setup in the laboratory.

show abstract

mentioning

confidence: 67%

mentioning

confidence: 99%

Online versus offline: The Web as a medium for response time data collection

Chetverikov

Upravitelev²

2015

Behav Res

View full text Add to dashboard Cite

show abstract

“…For specific paradigms, however, specialized software packages may be more suitable. For example, the Functional Measurement Experiment Builder Suite (Mairesse et al, 2008), for conducting experiments using the functional measurement paradigm, or WebExp (Keller, Subahshini, Mayo, & Corley, 2009), for conducting simple Web experiments, are both Java-based applications, and therefore are also platform independent, but they require a lot less prior programming knowledge.…”

Section: Limitations and Future Directionsmentioning

confidence: 99%

Tatool: A Java-based open-source programming framework for psychological studies

Bastian

Locher

Ruflin³

2012

Behav Res

View full text Add to dashboard Cite

show abstract

“…small differences in response time" (Skitka & Sargis, 2006, p. 547). Thus, although several well-established reaction time effects have been replicated in Web-based research (e.g., Crump, McDonnell, & Gureckis, 2013;Keller, Gunasekharan, Mayo, & Corley, 2009;Simcox & Fiez, 2014), skepticism remains widespread-for at least three reasons.…”

mentioning

confidence: 99%

Reaction time effects in lab- versus Web-based research: Experimental evidence

2015

View full text Add to dashboard Cite

Although Web-based research is now commonplace, it continues to spur skepticism from reviewers and editors, especially whenever reaction times are of primary interest. Such persistent preconceptions are based on arguments referring to increased variation, the limits of certain software and technologies, and a noteworthy lack of comparisons (between Web and lab) in fully randomized experiments. To provide a critical test, participants were randomly assigned to complete a lexical decision task either (a) in the lab using standard experimental software (E-Prime), (b) in the lab using a browser-based version (written in HTML and JavaScript), or (c) via the Web using the same browser-based version. The classical word frequency effect was typical in size and corresponded to a very large effect in all three conditions. There was no indication that the Web-or browser-based data collection was in any way inferior. In fact, if anything, a larger effect was obtained in the browser-based conditions than in the condition relying on standard experimental software. No differences between Web and lab (within the browser-based conditions) could be observed, thus disconfirming any substantial influence of increased technical or situational variation. In summary, the present experiment contradicts the still common preconception that reaction time effects of only a few hundred milliseconds cannot be detected in Web experiments.Keywords Web . Internet . Response latency . Reaction time . Word frequency effect Over the past two decades, research conducted online via the Internet has become increasingly frequent. Today, Web-based research is common across the whole range of social and behavioral sciences. This trend is not surprising, given the well-documented advantages of Web-based research, especially the possibility to recruit large, heterogeneous (and more representative) samples in less time and with lower costs than in traditional lab-or paper/pencil-based research (for overviews, see

show abstract

Timing accuracy of Web experiments: A case study using the WebExp software package

Cited by 93 publications

References 24 publications

Online versus offline: The Web as a medium for response time data collection

Online versus offline: The Web as a medium for response time data collection

Tatool: A Java-based open-source programming framework for psychological studies

Reaction time effects in lab- versus Web-based research: Experimental evidence

Contact Info

Product

Resources

About