Synch-Graph: Multisensory Emotion Recognition Through Neural Synchrony via Graph Convolutional Networks

Mansouri-Benssassi, Esma; Ye, Juan

doi:10.1609/aaai.v34i02.5491

Cited by 16 publications

(11 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…After this replacement, the accuracy of the video classifier is 56.8%. This is in line with state-of-the-art results in the literature on emotion recognition from RAVDESS videos, namely 57.5% with Synchronous Graph Neural Networks (8 emotions) [50]; 61% with ConvNet-LSTM (8 emotions) [1]; 59% with an RNN (7 emotions) [9], and 82.4% with stacked autoencoders (6 emotions) [5].…”

Section: A Dataset and Model Architecturesupporting

confidence: 88%

“…V, has been studied by other authors in-the-clear, i.e. without regards for privacy protection, using a variety of deep learning architectures, with reported accuracies in the 57%-82% range, depending on the number of emotion classes included in the study (6 to 8) [5], [50], [9], [1]. The ConvNet model that we trained for our experimental results in Sec.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Privacy-Preserving Video Classification with Convolutional Neural Networks

Pentyala¹,

Dowsley²,

Cock³

2021

Preprint

View full text Add to dashboard Cite

Many video classification applications require access to personal data, thereby posing an invasive security risk to the users' privacy. We propose a privacy-preserving implementation of single-frame method based video classification with convolutional neural networks that allows a party to infer a label from a video without necessitating the video owner to disclose their video to other entities in an unencrypted manner. Similarly, our approach removes the requirement of the classifier owner from revealing their model parameters to outside entities in plaintext.To this end, we combine existing Secure Multi-Party Computation (MPC) protocols for private image classification with our novel MPC protocols for oblivious single-frame selection and secure label aggregation across frames. The result is an end-to-end privacy-preserving video classification pipeline. We evaluate our proposed solution in an application for private human emotion recognition. Our results across a variety of security settings, spanning honest and dishonest majority configurations of the computing parties, and for both passive and active adversaries, demonstrate that videos can be classified with state-of-the-art accuracy, and without leaking sensitive user information.

show abstract

Section: A Dataset and Model Architecturesupporting

confidence: 88%

Section: Related Workmentioning

confidence: 99%

Privacy-Preserving Video Classification with Convolutional Neural Networks

Pentyala¹,

Dowsley²,

Cock³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…There are other components , such as Featureto-Label Level, Label-to-Label Level and Modality-to-Label Level, which work to pass messages among different type of nodes. Benssassi et al [101] synthesis the representation of neural synchrony graph from facial and speech features that are extracted via spiking neural networks(SNN). The neural synchrony graph is deemed as the input of Graph Convolutional Network(GCN) for emotion recognition.…”

Section: Multimodalitymentioning

confidence: 99%

Deep Emotion Recognition using Facial, Speech and Textual Cues: A Survey

Zhang¹,

Tan²

2023

Preprint

View full text Add to dashboard Cite

With the development of social media and human-computer interaction, it is essential to serve people by perceiving people's emotional state in videos. In recent years, a large number of studies tackle the issue of emotion recognition based on three most common modalities in videos, that is, face, speech and text. The focus of this paper is to sort out the relevant studies of emotion recognition using facial, speech and textual cues based on deep learning techniques due to the lack of review papers concentrating on the three modalities. In this paper, we firstly introduce widely accepted emotion models for the purpose of interpreting the definition of emotion. Then we introduce the state-of-the-art for emotion recognition based on unimodality including facial expression recognition, speech emotion recognition and textual emotion recognition. For multimodal emotion recognition, we summarize the feature-level and decision-level fusion methods in detail. In addition, the description of relevant benchmark datasets, the definition of metrics and the performance of the state-of-the-art in recent years are also outlined for the convenience of readers to find out the current research progress. Ultimately, we explore some potential research challenges and opportunities to give researchers reference for the enrichment of emotion recognition-related researches.

show abstract

“…There are other components , such as Feature-to-Label Level, Labelto-Label Level and Modality-to-Label Level, which work to pass messages among different type of nodes. Benssassi et al [98] synthesis the representation of neural synchrony graph from facial and speech features that are extracted via spiking neural networks(SNN). The neural synchrony graph is deemed as the input of Graph Convolutional Network(GCN) for emotion recognition.…”

Section: Multimodalitymentioning

confidence: 99%

Deep Emotion Recognition using Facial, Speech and Textual Cues: A Survey

Zhang¹,

Tan²

2021

Preprint

View full text Add to dashboard Cite

With the development of social media and human-computer interaction, video has become one of the most common data formats. As a research hotspot, emotion recognition system is essential to serve people by perceiving people’s emotional state in videos. In recent years, a large number of studies focus on tackling the issue of emotion recognition based on three most common modalities in videos, that is, face, speech and text. The focus of this paper is to sort out the relevant studies of emotion recognition using facial, speech and textual cues due to the lack of review papers concentrating on the three modalities. On the other hand, because of the effective leverage of deep learning techniques to learn latent representation for emotion recognition, this paper focuses on the emotion recognition method based on deep learning techniques. In this paper, we firstly introduce widely accepted emotion models for the purpose of interpreting the definition of emotion. Then we introduce the state-of-the-art for emotion recognition based on unimodality including facial expression recognition, speech emotion recognition and textual emotion recognition. For multimodal emotion recognition, we summarize the feature-level and decision-level fusion methods in detail. In addition, the description of relevant benchmark datasets, the definition of metrics and the performance of the state-of-the-art in recent years are also outlined for the convenience of readers to find out the current research progress. Ultimately, we explore some potential research challenges and opportunities to give researchers reference for the enrichment of emotion recognition-related researches.

show abstract

Synch-Graph: Multisensory Emotion Recognition Through Neural Synchrony via Graph Convolutional Networks

Cited by 16 publications

References 26 publications

Privacy-Preserving Video Classification with Convolutional Neural Networks

Privacy-Preserving Video Classification with Convolutional Neural Networks

Deep Emotion Recognition using Facial, Speech and Textual Cues: A Survey

Deep Emotion Recognition using Facial, Speech and Textual Cues: A Survey

Contact Info

Product

Resources

About