How to Run Behavioural Experiments Online: Best Practice Suggestions for Cognitive Psychology and Neuroscience

Abstract

The combination of a replication crisis, global COVID-19 pandemic, and recent technological advances have accelerated the on-going transition of research in cognitive psychology and behavioural neuroscience to the online realm. When participants cannot be tested in-person in the laboratory, data of acceptable quality can be collected online still. While conducting research online has many advantages and generic advice on their infrastructure and implementation exists, numerous pitfalls can hinder researchers addressing their research question appropriately or even result in unusable data. Here, we present detailed best practice suggestions that span the range from initial study design to the final interpretation of the data. These suggestions take a critical look at issues regarding the recruitment of typical and (sub)clinical samples, their comparison, and the importance of context dependency for each part of a study. We illustrate our suggestions by means of arecent online study investigating cognitive working memory skills in adult dyslexia.

Comparing Online Webcam- and Laboratory-Based Eye-Tracking for the Assessment of Infants’ Audio-Visual Synchrony Perception

“…Moreover, stimulus presentation timing in the online setting was not controlled for, but only assessed based on participant report (e.g., lags experienced in videos reported in the questionnaire). As there is a considerable variability in temporal precision between operating systems and browsers ( Gagné and Franzen, 2021 ; Mathôt and March, 2021 ), future studies can circumvent this issue by recording the participant’s screen and audio data. Yet, such recordings may add to the already high computational load on the participant’s device.…”

“…To conclude, our results indicate that online eye-tracking with infants is a promising avenue in developmental research and merits further exploration. However, the establishment of best practices for online data acquisition, data quality, and accuracy control, as well as analyses of data from a larger sample is essential (for a generic review, see Gagné and Franzen, 2021 ). Additionally, future studies aiming to assess the accuracy of online eye-tracking with adult and developmental populations could benefit from applying more challenging paradigms that require higher precision eye-tracking than preferential looking.…”

“…Sampling frequency (the number of times the eyes' positions are registered per second) also needs to be carefully contrasted between the two methods. While lab-based eye-tracking devices have a typical sampling rate of 500-1000 Hz, online sampling rates may only reach 30 Hz due to technical limitations of the participant's device and the eye-tracking algorithm itself (Gagné and Franzen, 2021).…”

“…However, methodological advances in online research with adults demonstrated that webcam-based eye-tracking systems can obtain data in comparable quality to data gathered in a traditional lab setting ( Xu et al, 2015 ; Papoutsaki et al, 2016 ; Bott et al, 2017 ; Semmelmann and Weigelt, 2018 ), and even smartphones can reach the accuracy of mobile eye-trackers ( Valliappan et al, 2020 ). Collecting eye-tracking data online entails higher variance, a lower sampling rate ( Gagné and Franzen, 2021 ), and increased experimental time, but shows no significant differences in spatial accuracy compared to in-lab recordings for adult data ( Semmelmann and Weigelt, 2018 ). Nonetheless, no research to date has directly compared webcam-based and in-lab eye-tracking data from infants.…”

“…Sampling frequency (the number of times the eyes’ positions are registered per second) also needs to be carefully contrasted between the two methods. While lab-based eye-tracking devices have a typical sampling rate of 500-1000 Hz, online sampling rates may only reach 30 Hz due to technical limitations of the participant’s device and the eye-tracking algorithm itself ( Gagné and Franzen, 2021 ). Missing data quantity or data loss (the relation between the expected number of gaze samples recorded by the eye-tracker and the actual number delivered) typically ranges from 2 to 20% in in-lab eye-trackers ( Cuve et al, 2021 ).…”