Emotion-Related Consciousness Detection in Patients With Disorders of Consciousness Through an EEG-Based BCI System

Pan, Jiahui; Xie, Qiuyou; Huang, Haiyun; He, Yanbin; Sun, Yeping; Yu, Rong; Li, Yuanqing

doi:10.3389/fnhum.2018.00198

Cited by 52 publications

(44 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In terms of the arousal space, higher frequency (beta, gamma) components from electrodes positioned on all positions have a higher correlation. Our results are partially consistent with the findings of previous work [35]. Pan et al found that the neural patterns had significantly higher responses in low frequencies at prefrontal sites and significantly higher theta and alpha responses at parietal sites for sad emotions and valence is a measure of both positive and negative emotion [35].…”

Section: Discussionsupporting

confidence: 93%

Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition

et al. 2019

Self Cite

View full text Add to dashboard Cite

Emotion recognition plays an essential role in human–computer interaction. Previous studies have investigated the use of facial expression and electroencephalogram (EEG) signals from single modal for emotion recognition separately, but few have paid attention to a fusion between them. In this paper, we adopted a multimodal emotion recognition framework by combining facial expression and EEG, based on a valence-arousal emotional model. For facial expression detection, we followed a transfer learning approach for multi-task convolutional neural network (CNN) architectures to detect the state of valence and arousal. For EEG detection, two learning targets (valence and arousal) were detected by different support vector machine (SVM) classifiers, separately. Finally, two decision-level fusion methods based on the enumerate weight rule or an adaptive boosting technique were used to combine facial expression and EEG. In the experiment, the subjects were instructed to watch clips designed to elicit an emotional response and then reported their emotional state. We used two emotion datasets—a Database for Emotion Analysis using Physiological Signals (DEAP) and MAHNOB-human computer interface (MAHNOB-HCI)—to evaluate our method. In addition, we also performed an online experiment to make our method more robust. We experimentally demonstrated that our method produces state-of-the-art results in terms of binary valence/arousal classification, based on DEAP and MAHNOB-HCI data sets. Besides this, for the online experiment, we achieved 69.75% accuracy for the valence space and 70.00% accuracy for the arousal space after fusion, each of which has surpassed the highest performing single modality (69.28% for the valence space and 64.00% for the arousal space). The results suggest that the combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources. The novelty of this work is as follows. To begin with, we combined facial expression and EEG to improve the performance of emotion recognition. Furthermore, we used transfer learning techniques to tackle the problem of lacking data and achieve higher accuracy for facial expression. Finally, in addition to implementing the widely used fusion method based on enumerating different weights between two models, we also explored a novel fusion method, applying boosting technique.

show abstract

Section: Discussionsupporting

confidence: 93%

Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…The correlation from the frontal head is slightly higher than that in posterior of the head. The findings are in line with the previous work [66].…”

Section: Preliminary Correlation Analysissupporting

confidence: 93%

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

2020

View full text Add to dashboard Cite

Human facial expressions are regarded as a vital indicator of one’s emotion and intention, and even reveal the state of health and wellbeing. Emotional states have been associated with information processing within and between subcortical and cortical areas of the brain, including the amygdala and prefrontal cortex. In this study, we evaluated the relationship between spontaneous human facial affective expressions and multi-modal brain activity measured via non-invasive and wearable sensors: functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) signals. The affective states of twelve male participants detected via fNIRS, EEG, and spontaneous facial expressions were investigated in response to both image-content stimuli and video-content stimuli. We propose a method to jointly evaluate fNIRS and EEG signals for affective state detection (emotional valence as positive or negative). Experimental results reveal a strong correlation between spontaneous facial affective expressions and the perceived emotional valence. Moreover, the affective states were estimated by the fNIRS, EEG, and fNIRS + EEG brain activity measurements. We show that the proposed EEG + fNIRS hybrid method outperforms fNIRS-only and EEG-only approaches. Our findings indicate that the dynamic (video-content based) stimuli triggers a larger affective response than the static (image-content based) stimuli. These findings also suggest joint utilization of facial expression and wearable neuroimaging, fNIRS, and EEG, for improved emotional analysis and affective brain–computer interface applications.

show abstract

“…The left superior temporal gyrus (STG, BA38) is dedicated to language processing, while the right side is considered the center of spatial awareness 55 . It is also important for social emotion processing 56 and facial emotion 57 . Interestingly, adolescents with a suicide attempt history show reduced grey-matter volume in the right BA38 58 – 60 .…”

Section: Discussionmentioning

confidence: 99%

EEG dynamics and neural generators of psychological flow during one tightrope performance

Leroy

Chéron

2020

Sci Rep

View full text Add to dashboard Cite

Psychological "flow" emerges from a goal requiring action, and a match between skills and challenge. Using high-density electroencephalographic (EEG) recording, we quantified the neural generators characterizing psychological "flow" compared to a mindful "stress" state during a professional tightrope performance. Applying swLORETA based on self-reported mental states revealed the right superior temporal gyrus (BA38), right globus pallidus, and putamen as generators of delta, alpha, and beta oscillations, respectively, when comparing "flow" versus "stress". Comparison of "stress" versus "flow" identified the middle temporal gyrus (BA39) as the delta generator, and the medial frontal gyrus (BA10) as the alpha and beta generator. These results support that "flow" emergence required transient hypo-frontality. Applying swLORETA on the motor command represented by the tibialis anterior EMG burst identified the ipsilateral cerebellum and contralateral sensorimotor cortex in association with on-line control exerted during both "flow" and "stress", while the basal ganglia was identified only during "flow". Over 40 years ago, Csikszentmihalyi 1 first introduced the concept of psychological "flow", defined as a singular mental state accompanying exceptional performance, which was later popularized as being "in the zone" 2. It is generally accepted that this exceptional state emerges from a clear goal that requires action, and a perfect match between specific skills and challenges 3-5. It is a unique sensation, accompanied by a transformation of time, which most commonly occurs in persons engaged in high-skill motor practices, such as champion athletes 6,7 and musicians 8-11. Psychological "flow" may also be considered a specific state of consciousness requiring involvement of the cortical areas participating in the neural correlates of consciousness (NCC). The NCC was initially defined by Crick and Koch 12 in 1990 as the minimal neuronal mechanisms jointly sufficient for any specific conscious experience, and the full NCC is considered the union of all content-specific NCCs 13,14. We hypothesized that flow identity may be an emergent conscious experience involving specific recruitment among the neural network of the NCC. This notion is supported by the idea that some brain functions necessitate an efficient network of connections and once such coordinated complexity is attained, the emergent properties of consciousness happen, producing the flow sensation. As this sensation goes along with or follows movement, a functional tradeoff between external and internal force must be continuously controlled. The sensation of flow would emerge in a particular physiological state where (1) an appropriate central resting state, including memorized items and motivation is present, (2) the initial intention must be translated by the descending motor commands to the muscles in order to generate forces and displacements, and (3) the ascending somesthetic signals must produce ideal feedback sensations closing the loop between action and sen...

show abstract

Emotion-Related Consciousness Detection in Patients With Disorders of Consciousness Through an EEG-Based BCI System

Cited by 52 publications

References 54 publications

Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition

Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

EEG dynamics and neural generators of psychological flow during one tightrope performance

Contact Info

Product

Resources

About