Day-to-day variability in hybrid, passive brain-computer interfaces: Comparing two studies assessing cognitive workload

Abstract

As hybrid, passive brain-computer interface systems become more advanced, it is important to grow our understanding of how to produce generalizable pattern classifiers of physiological data. One of the most difficult problems in applying machine learning algorithms to these data types is nonstationarity, which can evolve over the course of hours and days, and is more susceptible to changes resulting from complex cognitive function in comparison to simple, stimulus-based processes. This nonstationarity, referenced as day-to-day variability, results in the inability of many learning algorithms to generalize to new data. In previous work, we have shown that increasing the number of unique testing sessions used to form a learning set can improve the accuracy of classifying mental workload in a binary state paradigm. While this result was very promising, we did not address whether the additional discriminability was the result of a larger learning set or the uniqueness contributed by the testing sessions being spread over multiple days. Further, the simulation task used in this prior analysis was low-fidelity with respect to the task it attempted to model; whether these methods extend to more realistic task simulation environments has not been comparatively investigated. In this work, we compare these previous results to a second study, with a similar multi-day paradigm, that required participants to perform a more realistic simulation task. Comparative analysis of these two studies reveals that the improved generalization of the multi-day learning set is attributable, in large part, to the uniqueness of the multi-day paradigm. Further, this multi-day effect was also observed in the higher fidelity simulation study. These results help to validate the use of the multi-day learning set approach for improving overall system classification accuracy. Future studies should consider the use of multi-day designs for improving generalizability over other interesting dimensions.

Methods and Measures for Mental Stress Assessment in Surgery: A Systematic Review of 20 Years of Literature

“…Despite these advances and the surge in applying machine learning (ML) for mental stress assessment from multimodal datasets [51], the progress in implementing ML for objective stress and workload detection in the OR has been slower and limited to a few research groups [42], [44]. Challenges in quantifying mental states in real life [52], intra-and inter-subject variabilities [53], obtaining ontologies and annotating surgical stages [36], [54], privacy and data protection rights [55], and highly dynamic and case-dependent decisions and actions-interactions of surgical staff that result in data interpretation challenges [56] are a few contributing obstacles. These challenges are being addressed by new academic programs in digital health, data science, and AI-assisted technologies 1 and by public forums on data science and digital healthcare 2 .…”

Behavioral and Neural Variability of Naturalistic Arm Movements

“…Based on previous research (Derix et al, 2012;Dumas et al, 2010), we were also motivated to consider the effects of social interaction on neural activity. Finally, ECoG recordings are non-stationary (Klosterman et al, 2016;Yang et al, 2017), so we considered how movement-related neural activity varied over several hours and across recording days.…”

Behavioral and Neural Variability of Naturalistic Arm Movements

“…Based on previous research [81,82], we were also motivated to consider the effects of social interactions on neural activity. Finally, ECoG recordings are non-stationary [83][84][85], so we considered on how movement-related neural activity varied over several hours and across recording days.…”