Classification of Drowsiness Levels Based on a Deep Spatio-Temporal Convolutional Bidirectional LSTM Network Using Electroencephalography Signals

Jeong, Ji-Hoon; Yu, Baek Woon; Lee, Dae Hyeok; Lee, Seong Whan

doi:10.3390/brainsci9120348

Cited by 57 publications

(42 citation statements)

References 67 publications

(99 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…EEG is widely used in Brain-Computer Interfaces [ 24 ], as is highlighted by the 47 articles reviewed, here in which all of them have only used EEG, except three papers that used EEG combined with EOG [ 4 ], EEG with ElectroOculoGraphy (EOG), ElectroMyoGraphy (EMG), Skin Temperature (ST), Galvanic Skin Response (GSR), Blood Volume Pressure (BVP), Respiration Signal (RS) [ 25 ] and EEG plus EOG [ 26 ]. While EEG has proven to be a crucial tool in many domains, including BCI, it still suffers from some limitations that hamper its effectiveness due to its long pre and post-processing.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Hybrid Deep Learning (hDL)-Based Brain-Computer Interface (BCI) Systems: A Systematic Review

et al. 2021

View full text Add to dashboard Cite

Background: Brain-Computer Interface (BCI) is becoming more reliable, thanks to the advantages of Artificial Intelligence (AI). Recently, hybrid Deep Learning (hDL), which combines different DL algorithms, has gained momentum over the past five years. In this work, we proposed a review on hDL-based BCI starting from the seminal studies in 2015. Objectives: We have reviewed 47 papers that apply hDL to the BCI system published between 2015 and 2020 extracting trends and highlighting relevant aspects to the topic. Methods: We have queried four scientific search engines (Google Scholar, PubMed, IEEE Xplore and Elsevier Science Direct) and different data items were extracted from each paper such as the database used, kind of application, online/offline training, tasks used for the BCI, pre-processing methodology adopted, type of normalization used, which kind of features were extracted, type of DL architecture used, number of layers implemented and which optimization approach were used as well. All these items were then investigated one by one to uncover trends. Results: Our investigation reveals that Electroencephalography (EEG) has been the most used technique. Interestingly, despite the lower Signal-to-Noise Ratio (SNR) of the EEG data that makes pre-processing of that data mandatory, we have found that the pre-processing has only been used in 21.28% of the cases by showing that hDL seems to be able to overcome this intrinsic drawback of the EEG data. Temporal-features seem to be the most effective with 93.94% accuracy, while spatial-temporal features are the most used with 33.33% of the cases investigated. The most used architecture has been Convolutional Neural Network-Recurrent Neural Network CNN-RNN with 47% of the cases. Moreover, half of the studies have used a low number of layers to achieve a good compromise between the complexity of the network and computational efficiency. Significance: To give useful information to the scientific community, we make our summary table of hDL-based BCI papers available and invite the community to published work to contribute to it directly. We have indicated a list of open challenges, emphasizing the need to use neuroimaging techniques other than EEG, such as functional Near-Infrared Spectroscopy (fNIRS), deeper investigate the advantages and disadvantages of using pre-processing and the relationship with the accuracy obtained. To implement new combinations of architectures, such as RNN-based and Deep Belief Network DBN-based, it is necessary to better explore the frequency and temporal-frequency features of the data at hand.

show abstract

Section: Resultsmentioning

confidence: 99%

“…For example, some papers [ 10 , 11 , 57 ] obtained good performance, 98.81%, 95.33% and 92%, respectively, even though they did not use any preprocessing step. However, Jeong and colleagues and Saidutta and colleagues [ 26 , 58 ], using automated and advanced preprocessing, reached a performance of 87% and 81%, respectively.…”

Section: Discussionmentioning

confidence: 99%

Hybrid Deep Learning (hDL)-Based Brain-Computer Interface (BCI) Systems: A Systematic Review

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Recent BCI advances adopted deep learning techniques, which already yielded a dramatically high performance in other research fields, such as computer vision and natural language processing [33]. In particular, a few studies applied the deep networks to various BCI paradigms using EEG signals such as mental state detection [48]- [50], emotion recognition [51], [52], intention decoding using steady-state visual evoked potentials [16], P300 [53], [54], and MI [32], [34]- [36], [42]. Several studies for MI decoding using deep learning approaches focused on enhancing decoding performance for basic multi-classes (e.g., left hand, right hand, and foot) using a public dataset [35], [36].…”

Section: Discussionmentioning

confidence: 99%

EEG Classification of Forearm Movement Imagery Using a Hierarchical Flow Convolutional Neural Network

Jeong

Lee

et al. 2020

IEEE Access

Self Cite

View full text Add to dashboard Cite

Recent advances in brain-computer interface (BCI) techniques have led to increasingly refined interactions between users and external devices. Accurately decoding kinematic information from brain signals is one of the main challenges encountered in the control of human-like robots. In particular, although the forearm of an upper extremity is frequently used in daily life for high-level tasks, only few studies addressed decoding of the forearm movement. In this study, we focus on the classification of forearm movements according to elaborated rotation angles using electroencephalogram (EEG) signals. To this end, we propose a hierarchical flow convolutional neural network (HF-CNN) model for robust classification. We evaluate the proposed model not only with our experimental dataset but also with a public dataset (BNCI Horizon 2020). The grand-average classification accuracies of three rotation angles yield 0.73 (±0.04) for the motor execution (ME) task and 0.65 (±0.09) for the motor imagery (MI) task across ten subjects in our experimental dataset. Further, in the public dataset, the grand-averaged classification accuracies were 0.52 (±0.03) for ME and 0.51 (±0.04) for MI tasks across fifteen subjects. Our experimental results demonstrate the possibility of decoding complex kinematics information using EEG signals. This study will contribute to the development of a brain-controlled robotic arm system capable of performing high-level tasks. INDEX TERMS Brain-computer interface (BCI), electroencephalogram (EEG), convolutional neural network (CNN), forearm motor execution and motor imagery.

show abstract

“…30 EEG channels were placed on the subjects' scalp according to the international 10/20 system (Fp1, Fp2, F3, F4, Fz, FC1, FC2, FC5, FC6, T7, T8, C3, C4, Cz, CP1, CP2, CP5, CP6, TP9, TP10, P3, P4, P7, P8, Pz, PO9, PO10, O1, O2, and Oz) [29] as depicted in Fig. 1(b).…”

Section: B Experimental Environmentmentioning

confidence: 99%

“…Fatigue is commonly caused by a prolonged cognitive task, especially a repetitive or boring task [28], [29]. According to the British airline pilots' association (BALPA), 56% of the approximately 500 commercial pilots responded that they usually fell asleep during a flight.…”

Section: Introductionmentioning

confidence: 99%

Continuous EEG Decoding of Pilots’ Mental States Using Multiple Feature Block-Based Convolutional Neural Network

et al. 2020

Self Cite

View full text Add to dashboard Cite

Non-invasive brain-computer interface (BCI) has been developed for recognizing and classifying human mental states with high performances. Specifically, classifying pilots' mental states accurately is a critical issue because their cognitive states, which are induced by mental fatigue, workload, and distraction, may be fundamental in catastrophic accidents. In this study, we present an electroencephalogram (EEG) classification of four mental states (fatigue, workload, distraction, and the normal state) from EEG signals in both offline and pseudo-online analyses. To the best of our knowledge, this study is the first attempt to classify pilots' mental states using only EEG signals during continuous decoding. We recorded EEG signals from seven pilots under various simulated flight conditions. We proposed a multiple feature block-based convolutional neural network (MFB-CNN) with temporal-spatio EEG filters to recognize the pilot's current mental states. We validated the proposed method for two analyses across all subjects. In the offline analysis, we confirmed the classification accuracy of 0.75 (±0.04) and, in the pseudo-online analysis, we obtained the detection accuracy of 0.72 (±0.20), 0.72 (±0.27), and 0.61 (±0.18) for fatigue, workload, and distraction, respectively. Hence, we demonstrate the feasibility of classifying various types of mental states for implementation in real-world environments. INDEX TERMS Brain-computer interface (BCI), Electroencephalogram (EEG), Mental states, Deep convolutional neural network.

show abstract

Classification of Drowsiness Levels Based on a Deep Spatio-Temporal Convolutional Bidirectional LSTM Network Using Electroencephalography Signals

Cited by 57 publications

References 67 publications

Hybrid Deep Learning (hDL)-Based Brain-Computer Interface (BCI) Systems: A Systematic Review

Hybrid Deep Learning (hDL)-Based Brain-Computer Interface (BCI) Systems: A Systematic Review

EEG Classification of Forearm Movement Imagery Using a Hierarchical Flow Convolutional Neural Network

Continuous EEG Decoding of Pilots’ Mental States Using Multiple Feature Block-Based Convolutional Neural Network

Contact Info

Product

Resources

About