Development of a Real-Time Emotion Recognition System Using Facial Expressions and EEG based on machine learning and deep neural network methods

Hassouneh, Aya; Mutawa, A. M.; Murugappan, M.

doi:10.1016/j.imu.2020.100372

Cited by 196 publications

(69 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Understanding emotional signals in everyday life environments becomes an important aspect that influences people's communication through verbal and nonverbal behavior [ 40 ]. One such example of emotional signals is expressed through facial expression which is known to be one of the most immediate means of human beings to communicate their emotions and intentions [ 41 ]. With the advancement of technologies in brain-computer interface and neuroimaging, it is now feasible to capture the brainwave signals nonintrusively and to measure or control the motions of devices virtually [ 42 ] or physically such as wheelchairs [ 43 ], mobile phone interfacing [ 44 ], or prosthetic arms [ 45 , 46 ] with the use of a wearable EEG headset.…”

Section: Emotionsmentioning

confidence: 99%

EEG-Based Emotion Recognition: A State-of-the-Art Review of Current Trends and Opportunities

Suhaimi

Mountstephens

Teo

2020

Computational Intelligence and Neuroscience

243

View full text Add to dashboard Cite

Emotions are fundamental for human beings and play an important role in human cognition. Emotion is commonly associated with logical decision making, perception, human interaction, and to a certain extent, human intelligence itself. With the growing interest of the research community towards establishing some meaningful “emotional” interactions between humans and computers, the need for reliable and deployable solutions for the identification of human emotional states is required. Recent developments in using electroencephalography (EEG) for emotion recognition have garnered strong interest from the research community as the latest developments in consumer-grade wearable EEG solutions can provide a cheap, portable, and simple solution for identifying emotions. Since the last comprehensive review was conducted back from the years 2009 to 2016, this paper will update on the current progress of emotion recognition using EEG signals from 2016 to 2019. The focus on this state-of-the-art review focuses on the elements of emotion stimuli type and presentation approach, study size, EEG hardware, machine learning classifiers, and classification approach. From this state-of-the-art review, we suggest several future research opportunities including proposing a different approach in presenting the stimuli in the form of virtual reality (VR). To this end, an additional section devoted specifically to reviewing only VR studies within this research domain is presented as the motivation for this proposed new approach using VR as the stimuli presentation device. This review paper is intended to be useful for the research community working on emotion recognition using EEG signals as well as for those who are venturing into this field of research.

show abstract

Section: Emotionsmentioning

confidence: 99%

EEG-Based Emotion Recognition: A State-of-the-Art Review of Current Trends and Opportunities

Suhaimi

Mountstephens

Teo

2020

Computational Intelligence and Neuroscience

243

View full text Add to dashboard Cite

show abstract

“…Proposed method for speech signal recognition system based on EMD and non-linear features: a simplified representation. b Detailed illustration of different processing steps involved in proposed SER system frequency scales[29]. Hence, predefined selection of any IMF component or frequency scale will lead to a loss of information.…”

mentioning

confidence: 99%

Emotion classification from speech signal based on empirical mode decomposition and non-linear features

Krishnan

Raj

Rajangam

2021

Complex Intell. Syst.

View full text Add to dashboard Cite

Emotion recognition system from speech signal is a widely researched topic in the design of the Human–Computer Interface (HCI) models, since it provides insights into the mental states of human beings. Often, it is required to identify the emotional condition of the humans as cognitive feedback in the HCI. In this paper, an attempt to recognize seven emotional states from speech signals, known as sad, angry, disgust, happy, surprise, pleasant, and neutral sentiment, is investigated. The proposed method employs a non-linear signal quantifying method based on randomness measure, known as the entropy feature, for the detection of emotions. Initially, the speech signals are decomposed into Intrinsic Mode Function (IMF), where the IMF signals are divided into dominant frequency bands such as the high frequency, mid-frequency , and base frequency. The entropy measures are computed directly from the high-frequency band in the IMF domain. However, for the mid- and base-band frequencies, the IMFs are averaged and their entropy measures are computed. A feature vector is formed from the computed entropy measures incorporating the randomness feature for all the emotional signals. Then, the feature vector is used to train a few state-of-the-art classifiers, such as Linear Discriminant Analysis (LDA), Naïve Bayes, K-Nearest Neighbor, Support Vector Machine, Random Forest, and Gradient Boosting Machine. A tenfold cross-validation, performed on a publicly available Toronto Emotional Speech dataset, illustrates that the LDA classifier presents a peak balanced accuracy of 93.3%, F1 score of 87.9%, and an area under the curve value of 0.995 in the recognition of emotions from speech signals of native English speakers.

show abstract

“…There is no standard number of AUs have been proposed yet. The total number of AUs and AUs’ locations is based on the application and its requirement [ 28 , 29 ]. The distance between the AUs is mostly referred to as a standard feature for facial expression recognition.…”

Section: Related Workmentioning

confidence: 99%

“…A novel vectorized emotion recognition model is proposed to identify three primary emotions: angry, happy, and neutral, using 70 facial vectors and a deep neural network (DNN), and achieved a mean accuracy of 84.33% [ 39 ]. In recent literature, the researchers have used spatial and temporal information from input video sequences to classify different facial expressions using CNN, Ensemble Multi-level CNN, and Long Short-term Memory (LSTM) [ 29 , 40 – 43 ]. Some of the common issues reported in the earlier works due to the lack of samples or data sets, low accuracy in classifying facial expressions, higher computational complexity (more memory and power required for processing the data), not suitable for real-time applications, and not user-friendly approach (restrictions in using the system for a variety of applications) [ 44 ].…”

Section: Related Workmentioning

confidence: 99%

Facial geometric feature extraction based emotional expression classification using machine learning algorithms

Murugappan

Mutawa

2021

PLoS ONE

View full text Add to dashboard Cite

Emotion plays a significant role in interpersonal communication and also improving social life. In recent years, facial emotion recognition is highly adopted in developing human-computer interfaces (HCI) and humanoid robots. In this work, a triangulation method for extracting a novel set of geometric features is proposed to classify six emotional expressions (sadness, anger, fear, surprise, disgust, and happiness) using computer-generated markers. The subject’s face is recognized by using Haar-like features. A mathematical model has been applied to positions of eight virtual markers in a defined location on the subject’s face in an automated way. Five triangles are formed by manipulating eight markers’ positions as an edge of each triangle. Later, these eight markers are uninterruptedly tracked by Lucas- Kanade optical flow algorithm while subjects’ articulating facial expressions. The movement of the markers during facial expression directly changes the property of each triangle. The area of the triangle (AoT), Inscribed circle circumference (ICC), and the Inscribed circle area of a triangle (ICAT) are extracted as features to classify the facial emotions. These features are used to distinguish six different facial emotions using various types of machine learning algorithms. The inscribed circle area of the triangle (ICAT) feature gives a maximum mean classification rate of 98.17% using a Random Forest (RF) classifier compared to other features and classifiers in distinguishing emotional expressions.

show abstract

Development of a Real-Time Emotion Recognition System Using Facial Expressions and EEG based on machine learning and deep neural network methods

Cited by 196 publications

References 10 publications

EEG-Based Emotion Recognition: A State-of-the-Art Review of Current Trends and Opportunities

EEG-Based Emotion Recognition: A State-of-the-Art Review of Current Trends and Opportunities

Emotion classification from speech signal based on empirical mode decomposition and non-linear features

Facial geometric feature extraction based emotional expression classification using machine learning algorithms

Contact Info

Product

Resources

About