Conducting linguistic experiments online with OpenSesame and OSWeb

Abstract

In this Methods Showcase, we outline a workflow for running behavioral experiments online, with a focus on linguistic experiments. The workflow that we describe here relies on three tools: OpenSesame/ OSWeb (open source) provides a user-friendly graphical interface for developing experiments; JATOS (open source) is server software for hosting experiments; and Prolific (commercial) is a platform for recruiting participants. These three tools integrate well with each other, and together they provide a workflow that requires little technical expertise. We discuss several challenges that are associated with running online experiments, including temporal precision, the ability to implement counterbalancing, and data quality. We conclude that these problems are real but surmountable, and that in many cases online experiments are a viable alternative to laboratory-based experiments.

Comparing Online Webcam- and Laboratory-Based Eye-Tracking for the Assessment of Infants’ Audio-Visual Synchrony Perception

“…Moreover, stimulus presentation timing in the online setting was not controlled for, but only assessed based on participant report (e.g., lags experienced in videos reported in the questionnaire). As there is a considerable variability in temporal precision between operating systems and browsers ( Gagné and Franzen, 2021 ; Mathôt and March, 2021 ), future studies can circumvent this issue by recording the participant’s screen and audio data. Yet, such recordings may add to the already high computational load on the participant’s device.…”

How to Run Behavioural Experiments Online: Best Practice Suggestions for Cognitive Psychology and Neuroscience

“…This refresh rate is commonly 60 Hz, which only allows for presentation durations of multiples of 16.6 ms. As well, if a frame glitches and is omitted such short presentation durations are substantially shortened. However, timing concerns for longer stimulus presentation durations on modern browsers, and with the optimal operating system / browser pairing, have been alleviated in recent years (Anwyl-Irvine, Dalmaijer, et al, 2020;Bridges et al, 2020;Gallant & Libben, 2019;Mathôt & March, 2021;Pronk et al, 2020). Timing accuracy of visual displays and reaction times tend to be best when chromium-based browsers are being used, independent of the operating system (e.g., Google Chrome; Anwyl-Irvine, Dalmaijer, et al, 2020).…”

“…An efficient online workflow is also important for success, which is discussed at length elsewhere (Grootswagers, 2020;Mathôt & March, 2021;Sauter et al, 2020). The workflow of the presented experiment included the creation of our problems can be presented as error messages before the end of the experiment to avoid participants spending a long time wrongly doing the task and the researchers having to award credit or pay participants for unusable data.…”

“…Online research provides solutions for these issues that can be implemented with relative ease (see advantages listed in Table 1). These solutions include the important ability to efficiently collect large samples from a wider socio-economic background for less monetary and time investments on the researchers' end, which would otherwise be difficult to obtain in-person (for discussions, see Grootswagers, 2020;Mathôt & March, 2021;Sauter et al, 2020), and all-in-one platforms offering experiment building, presentation, recruitment and distribution capabilities. Hence, online research could become advantageous for both single studies and research fields in the context of this replication crisis.…”

Web-based language production experiments: Semantic interference assessment is robust for spoken and typed response modalities

“…Latencies of key presses can be detected with basic methods, and are, next to mouse clicks, the main means of reaction time detection in both labbased and online experiments. They have been implemented in a wide variety of online experiments, including psycholinguistic experiments using categorization tasks (Mathôt & March, 2021;Vogt et al, 2021). Additionally, it has already been shown that typewritten texts can be preprocessed automatically and a wide range of different procedures exists, even controlling for typing errors (Borrie et al, 2019;Bosker, 2021;Navarro, 2001).…”