Multilevel Feature Fusion With 3D Convolutional Neural Network for EEG-Based Workload Estimation

Abstract

Mental workload is defined as the proportion of the information processing capability used to perform a task. High cognitive load requires additional resources to process information; this demand for additional resources may reduce the processing efficiency and performance. Therefore, the technique of workload estimation can ensure a proper working environment to promote the working efficiency of each person. In this paper, we propose a three-dimensional convolutional neural network (3D CNN) employing a multilevel feature fusion algorithm for mental workload estimation using electroencephalogram (EEG) signals. The 1D EEG signals are converted to 3D EEG images to enable the 3D CNN to learn the spectral and spatial information over the scalp. The multilevel feature fusion framework integrates local and global neuronal activities by workload tasks in the 3D CNN algorithm. Multilevel features are extracted in each layer of the 3D convolution operation and each multilevel feature is multiplied by a weighting factor, which determines the importance of the feature. The weighting factor is adaptively estimated for each EEG image by a backpropagation process. Furthermore, we generate subframes from each EEG image and propose a temporal attention technique based on the long short-term memory model (LSTM) to extract a significant subframe at each multilevel feature that is strongly correlated with task difficulty. To verify the performance of our network, we performed the Sternberg task to measure the mental workload of the participant, which was classified according to its difficulty as low or high workload condition. We showed that the difficulty of the workload was well designed, which was reflected in the behavior of the participant. Our network is trained on this dataset and the accuracy of our network is 90.8 %, which is better than that of conventional algorithms. We also evaluated our method using the public EEG dataset and achieved 93.9 % accuracy. INDEX TERMS Convolutional neural network, electroencephalogram (EEG), feature fusion, mental workload, working memory.

Tracking of Mental Workload with a Mobile EEG Sensor

“…Importantly, the accuracy reported for cross-task classification is rather low, which is in line with the high data variability we observed. More recent publications suggest that larger data sets in combination with more complex deep-learning algorithms, such as recurrent and convolutional networks or even their combinations [ 45 , 46 ] may be promising alternatives that need further investigation. Moreover, deep learning algorithms are a promising candidate in terms of handling cross-task classification [ 47 ].…”

FGANet: fNIRS-Guided Attention Network for Hybrid EEG-fNIRS Brain-Computer Interfaces

“…Therefore, to include spatiotemporal information in the input data for training the network, we converted 1D EEG signals to 3D EEG tensors. To obtain 3D EEG images, we projected the 3D electrode locations on the scalp into a 2D image with size 16 × 16 using azimuthal equidistant projection as in [34] (Fig. 3(a)).…”

“…Because 3D convolution can extract both spatial and temporal information, 3D CNNs achieve superior performance in various areas where input data contain 3D data, such as video recognition and EEG decoding [34], [35], [49], [50]. Therefore, we constructed ESNet and FSNet consisting of three 3D convolutional layers with a rectified linear unit (ReLU) activation function to extract spatiotemporal information from 3D EEG and fNIRS tensors, respectively.…”

EEG Correlates of Driving Performance

“…For example, SVC's have been used to classify different memory workloads, using the n-back task [38], with excellent results. In addition CNN's have been shown to have high classification accuracies for neuroscience applications, where a typical processing chain will convert 1-D EEG signals to 3D EEG images and enable a 3D CNN to learn the spectral and spatial information over the scalp [39]. However, these techniques are considered to be black-box techniques, which generally lack transparency and explainability, limiting psychological interpretations and often failing to progress the knowledge space, which is a fundamental requisite of neuroscience.…”