Emotion Recognition in Conversations Using Brain and Physiological Signals

Saffaryazdi, Nastaran; Goonesekera, Yenushka; Saffaryazdi, Nafiseh; Hailemariam, Nebiyou Daniel; Temesgen, Ebasa Girma; Nanayakkara, Suranga; Broadbent, Elizabeth; Billinghurst, Mark

doi:10.1145/3490099.3511148

Cited by 20 publications

(10 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Regarding the third hypothesis (emotional hypothesis), analysis of the results obtained using the statistical method of covariance showed that the strategic training of NLP increased learners' EI. The result is consistent with the research results (Gardner and Stough, 2002;Keezhatta and Omar, 2019;Mitsea, 2020, 2021;Resmisari and Sitepu, 2022;Saffaryazdi et al, 2022).…”

Section: Discussionsupporting

confidence: 92%

“…Five scopes of EI, such as self-awareness, self-regulation, self-motivation, empathy, and social ability, were categorized (Resmisari and Sitepu, 2022 ). Based on the NLP methods, people can handle emotional challenges when coping with difficulties (Weare and Gray, 2003 ; Saffaryazdi et al, 2022 ). Various ways have been considered in NLP to increase each dimension of EI, such as for self-awareness (values hierarchy and goal setting), self-regulation (dissociative technique), self-motivation (associative approach), empathy (matching and mirroring), for social skills (rapport).…”

Section: Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

The effect of neurolinguistic programming on academic achievement, emotional intelligence, and critical thinking of EFL learners

Zhang¹,

Davarpanah

Izadpanah

2023

Front. Psychol.

View full text Add to dashboard Cite

Neurolinguistic programming (NLP) is a method of personal communication. This study aimed to determine the effect of NLP strategies on academic achievement, emotional intelligence, and critical thinking. Although NLP has been studied, more studies still need to be conducted on this variable contributing to language learning success. This experimental study was conducted with a pretest-posttest design with the control group in 2021. Sampling was conducted through the multistage cluster random sampling (MCRS) method, and based on the Cambridge placement test (2010), 50 students proved to be at an advanced level and participated in this study. To test the hypotheses, an ANCOVA test was employed. Participants were randomly divided into two control (25 people) and experimental groups (25 people). They were performed on the experimental group during 12 sessions of 90 min of the strategic training in NLP. In the experimental group, the mean and std of critical thinking was 16.24 ± 2.59 in the pretest, which increased to 18.88 ± 2.77 in the posttest; the mean and std of academic achievement was 155.02 ± 15.90 in the pretest, which rose to 171.70 ± 10.83 in the posttest and the mean and std of emotional intelligence was 96.51 ± 12.44 in the pretest, which increased to 118.28 ± 6.18 in the posttest. The results of data analysis by covariance method showed that NLP was practical on learners' academic achievement, emotional intelligence, and critical thinking. Justifications and implications for the study's findings and suggestions for further research are presented.

show abstract

Section: Discussionsupporting

confidence: 92%

Section: Literature Reviewmentioning

confidence: 99%

The effect of neurolinguistic programming on academic achievement, emotional intelligence, and critical thinking of EFL learners

Zhang¹,

Davarpanah

Izadpanah

2023

Front. Psychol.

View full text Add to dashboard Cite

show abstract

“…For arousal and valence classification, accuracy of 63.06% and 62.41% was achieved, respectively, using peripheral signals representation, and 93.06% and 91.95% by combining both EEG and peripheral representation, accordingly. Saffaryazdi et al [34] presented a study for recognizing emotions in a face-to-face conversation using hand-crafted features from EEG, GSR, and PPG signals. Classification results achieved F-score of 77.6% and 80% for arousal and valence, respectively, in subject-dependent tests, while 68% and 64% where achieved in subject-independent tests.…”

Section: B Brain Signals-based Methodsmentioning

confidence: 99%

LSTM-Modeling of Emotion Recognition Using Peripheral Physiological Signals in Naturalistic Conversations

Zitouni

Park

Lee

et al. 2023

IEEE J. Biomed. Health Inform.

View full text Add to dashboard Cite

The automated recognition of human emotions plays an important role in developing machines with emotional intelligence. Major research efforts are dedicated to the development of emotion recognition methods. However, most of the affective computing models are based on images, audio, videos and brain signals. Literature lacks works that focus on utilizing only peripheral signals for emotion recognition (ER), which can be ideally implemented in daily life settings. Therefore, this paper present a framework for ER on the arousal and valence space, based on using multi-modal peripheral signals. The data used in this work were collected during a debate between two people using wearable devices. The emotions of the participants were rated by multiple raters and converted into classes in correspondence to the arousal and valence space. The use of a dynamic threshold for ratings conversion was investigated. An ER model is proposed that uses a Long Short-Term Memory (LSTM)-based architecture for classification. The model uses heart rate (HR), temperature (T), and electrodermal activity (EDA) signals as its inputs with emotional cues. Additionally, a post-processing prediction mechanism is introduced to enhance the recognition performance. The model is implemented to study the use of individual and different combinations of the peripheral signals, as well as utilizing annotations from different ratings. Additionally, it is employed for classification of valence and arousal in an independent and combined fashion, under subject dependent and independent scenarios. The experimental results have justified the efficient performance of the proposed framework, achieving classification accuracy >96% and >93% for the independent and

show abstract

“…In state‐of‐the‐art studies, the multisensory approach has been used to assess mental workload, identify emotional states, and analyze the learning and information processing process. For instance, Campisi et al [ 12 ] investigated the mental workload of students while web browsing using augmented reality, Mutlu‐Bayraktar et al [ 13 ] examined the effects of split attention in multimedia learning environments using eye‐tracking and EEG sensors, Saffaryazdi et al [ 14 ] analyzed people's emotional state during stimuli exposure using GSR and PPG sensors, and Srivastava et al [ 15 ] investigated the process of information learning and visual attention using eye‐trackers.…”

Section: State‐of‐the‐art Wearable Devices For Mind‐wandering Detectionmentioning

confidence: 99%

Exploring the Elusive Mind: A Multimodal Wearable Sensor Solution for Measuring Mind Wandering in University Students

Khosravi,

Li,

Khan

et al. 2023

Advanced Sensor Research

View full text Add to dashboard Cite

Mind‐wandering is a typical daily phenomenon during which attention shifts from external stimuli to internal trains of thought. It affect students' learning by impairing comprehension, diminishing academic achievement, impeding critical thinking, and encouraging a lack of attention and engagement in classroom activities. This study aims to introduce a new method of detecting and tracking mind wandering in university students. This approach involves using wearable sensors, including galvanic skin response (GSR), photoplethysmography (PPG), and eye‐trackers, along with machine learning techniques. The study provides a proof of concept for this multisensory approach. The association between longer fixation duration and mind wandering, and the influence of an instructor's presence on fixation allocation, and, consequently, the frequency and occurrence of mind wandering is investigated. Furthermore, the feasibility of using eye‐trackers in conjunction with GSR and PPG sensors for detecting mind wandering through a wearable multisensory data collection system is assessed. The wearable multisensory device is evaluated by ten participants (university students, males/females aged between 21‐30). Two distinct machine learning methods, support vector machine (SVM) and gated recurrent unit (GRU), are used as classification models. With sensor fusion, the SVM and GRU models yielded maximum accuracies of 86.53% and 89.86%, respectively. Moreover, participants are observed to fixate on instructors more often, just before instances of mind wandering.

show abstract

Emotion Recognition in Conversations Using Brain and Physiological Signals

Cited by 20 publications

References 63 publications

The effect of neurolinguistic programming on academic achievement, emotional intelligence, and critical thinking of EFL learners

The effect of neurolinguistic programming on academic achievement, emotional intelligence, and critical thinking of EFL learners

LSTM-Modeling of Emotion Recognition Using Peripheral Physiological Signals in Naturalistic Conversations

Exploring the Elusive Mind: A Multimodal Wearable Sensor Solution for Measuring Mind Wandering in University Students

Contact Info

Product

Resources

About