Online Timing Accuracy and Precision: A comparison of platforms, browsers, and participant's devices

Abstract

Due to its increasing ease-of-use and ability to quickly collect large samples, online behavioral research is currently booming. With this increasing popularity, it is important that researchers are aware of who online participants are, and what devices and software they use to access experiments. While it is somewhat obvious that these factors can impact data quality, it remains unclear how big this problem is. To understand how these characteristics impact experiment presentation and data quality, we performed a battery of automated tests on a number of representative setups. We investigated how different web-building platforms (Gorilla, jsPsych, Lab.js, and psychoJS/PsychoPy3), browsers (Chrome, Edge, Firefox, and Safari), and operating systems (mac OS and Windows 10) impact display time across 30 different frame durations for each software combination. In addition, we employed a robot actuator in representative setups to measure response recording across aforementioned platforms, and between different keyboard types (desktop and integrated laptop). We then surveyed over 200 000 participants on their demographics, technology, and software to provide context to our findings. We found that modern web-platforms provide a reasonable accuracy and precision for display duration and manual response time, but also identify specific combinations that produce unexpected variance and delays. While no single platform stands out as the best in all features and conditions, our findings can help researchers make informed decisions about which experiment building platform is most appropriate in their situation, and what equipment their participants are likely to have.

Building, Hosting and Recruiting: A Brief Introduction to Running Behavioral Experiments Online

“…Notably, these means are inflated by particularly bad performance using the Safari browser and Mac OS X. The offline-based comparison, PsychoPy and Opensesame achieved precisions of 1 ms to 4 ms, with only minor exceptions [60,61], most notably with audio playback.…”

“…Additionally, modern screen refresh rates are almost exclusively set to 60 Hz (de facto standard), making certain specifications of online studies a bit more predictable. Among others [57][58][59], two recent large studies [60,61] investigated timing precision (unintended variability in stimulus presentation) of several online and offline solutions. The online-based comparison found good overall precision for Gorilla (13 ms), jsPsych (26 ms), PsychoJS (−6 ms) and lab.js (10 ms).…”

A primer on running human behavioural experiments online

“…In general, latencies and variabilities are higher in web-based compared to lab-environments. Several studies have assessed the quality of timing in online studies, with encouraging results (Anwyl-Irvine, Dalmaijer, et al, 2020;Bridges et al, 2020;Pronk et al, 2019;Reimers & Stewart, 2015). An online evaluation of a masked priming experiment showed that very short stimulus durations (i.e., under 50ms) can be problematic (but see Barnhoorn et al, 2014), but other classic experimental psychology paradigms that rely on reaction times (e.g., Stroop, flanker, and Simon tasks)…”

Detection and Recognition of Fearful Facial Expressions During the Coronavirus Disease (COVID-19) Pandemic in an Italian Sample: An Online Experiment

“…Indeed, when an experimental task is run online, technical criticisms (that cannot be solved remotely) in terms of timing (such as the accurate timing of visual stimuli presentation, or of the subjective responses) and mostly related to the participants' bandwidth should be considered. The discussion of such timing issues is out of the scope of the present manuscript; however, further comments on the constraints of online behavioral tasks were reported by Crump et al (2013) and, more recently, by Anwyl-Irvine et al (2020). Because of the criticisms on the RT, it is highly recommendable to rate the individual's performance accordingly to an index (such as the percentage of the level of Accuracy) registered independently from the timing.…”