How Does Day-to-Day Variability in Psychophysiological Data Affect Classifier Accuracy?

Abstract

Combined psychophysiological measures have been used to determine mental workload in operators, but the day-today reliability of these measures has not been determined. Data were collected four times over a one month period. Two classifiers were trained with these data and their ability to correctly discriminate between two levels of task difficulty with new data was tested. Both classifiers very accurately discriminated between the two levels of task difficulty using data collected on the same day as the training data. However, the accuracy was considerably reduced when tested on data from days different from the training data. The implications for application of these procedures are discussed.

Estimation of task workload from EEG data: New and current tools and perspectives

“…Linear classifiers [27,28,29] have been shown to be good contenders, and are widely appreciated for to their speed, simplicity, and interpretability. Nevertheless, nonlinear classifiers can potentially capture richer relationships between signal features and cognitive state, as has been demonstrated empirically in some scenarios [8,30]. As our data is high-dimensional, we use a feature-selecting nonlinear classifier.…”

“…We contrast methods that extract spectral features directly from sensor signals [8] with methods that extract spectral features from estimated (brain) source signals [16,17]. Since EEG sensor signals are a linear mixture of source signals conveyed to the electrodes by volume conduction, this amounts to a choice of a linear spatial filter and a method to determine the filter's parameters [18,19].…”

“…Most approaches for real-time mental state estimation from the EEG (or other physiological measures) of a person can be understood as signal processing systems that translate biosignals into a time-varying (and often binary) output [12], in this case a workload index [8,10]. In recent years, these and other cognitive monitoring systems have adopted a (supervised) machine learning approach to adaptively learn person-specific and session-specific parameters from calibration data [5,12,13].…”

“…There is an increasing need to compare and evaluate cognitive state estimation methods on equal footing, in particular because of the great variety of experimental tasks that have been proposed to assess different aspects of Manuscript workload [8,9,10]. The ever-present risk of circular analysis in complex pattern recognition problems [11] also demands reliable and agreed-upon evaluation procedures for measuring estimation performance.…”

“…Workload measurement technology has been incrementally improved and tuned to the point where claims of near-perfect accuracy are not unheard of, despite relatively high recording noise levels, tremendous complexity of the brain, and current incomplete understanding of the underlying brain EEG signals [7,8].…”