Seeking reproducibility: Assessing a multimodal study of the testing effect

Abstract

Low‐cost devices have widened the use of multimodal data in experiments providing a more complete picture of behavioural effects. However, the accurate collection and combination of multimodal and behavioural data in a manner that enables reproducibility is challenging and often requires researchers to refine their approaches. This paper presents a direct replication of a multimodal wordlist experiment. Specifically, we use a low‐cost Emotiv EPOC® to acquire electrophysiological measures of brain activity to investigate whether retrieval during learning facilitates the encoding of subsequent learning as measured by performance on recall tests and reflected by changes in alpha wave oscillations. Behavioural results of the wordlist experiment were replicated, but physiological results were not. We conclude the paper by highlighting the challenges faced in terms of replicating the previous work and in attempting to facilitate the reproducibility of our own experiment.

Building pipelines for educational data using AI and multimodal analytics: A “grey‐box” approach

“…The physiological data sources include eye-tracking, electroencephalography, facial features and arousal data (HR, blood volume pressure (BVP), electrodermal activity (EDA) and skin temperature). Various combinations of such data sources have been used in the past to explain (Raca & Dillenbourg, 2014) and/or predict (Beardsley, Hernández-Leo, & Ramirez-Melendez, 2018) learning behaviors (Furuichi & Worsley, 2018) and/or performance (Junokas, Lindgren, Kang, & Morphew, 2018).…”

“…Furthermore, considerable amount of research has also been conducted to predict learning performance in diverse learning tasks, using multimodal data. Specifically, researchers have used EEG and behavioral data (eg, reaction time from clickstreams, Beardsley et al , ) to predict students' recall, or gestures, postures and body movements to predict students' performance in repeating, recalling and association tasks (Junokas et al , ). In a project‐based learning case, Spikol et al () used objects created by the students in their respective projects, in combination with students' positions, hand gestures, facial expressions, audio, video and interaction patterns with the physical computing platform, aiming to predict the quality and correctness of the solution.…”

Temporal analysis of multimodal data to predict collaborative learning outcomes

“…Temporal collaborative multimodal data 1529 motivation). For example, to predict students' performance in terms of recall, quality, correctness or self-assessment, researchers have used different combinations of brain, behavioral or body signals, as well as audio cues and learning artifacts (Beardsley, Hernández-Leo, & Ramirez-Melendez, 2018;Chen et al, 2016;Di Mitri et al, 2017;Junokas, Lindgren, Kang, & Morphew, 2018;Spikol, Ruffaldi, Dabisias, & Cukurova, 2018). Combinations of different data streams yielded accurate predictions for a range of performance measures.…”

“…As can be inferred from the above examples, the data modalities used in individual and collaborative MMLA range greatly with log data, clickstreams, audio, video, dialogs, facial expressions, gestures, posture, motion, gaze and biological being most common (Beardsley et al, 2018;Chen et al, 2016;Junokas et al, 2018;Liu & Stamper, 2017;Mattingly et al, 2019;Smith et al, 2016;Spikol et al, 2018). From these data streams, a wide range of higher level features can be inferred including affect, attention, cognitive processing, stress and fatigue.…”

Interdisciplinary Open Science: What are the implications for educational technology research?

“…The current landscape of Open Science in Education includes some evidence of emergent discourse and changing practice. This includes two special issues in journals that have put out a call for research papers on the subject, with one invited review paper (see van der Zee & Reich, 2018), and an explicit attempt at seeking reproducibility (Beardsley, Hernández-Leo, & Ramirez-Melendez, 2018), notwithstanding the adoption of some Open Science practices by educational researchers on the ground (e.g., pre-registered research plan, MacQuarrie et al, 2018; pre-print, Selwyn, 2017). The British Journal of Educational Psychology has also recently (13 July 2018) announced that they are now accepting registered reports.…”