Audio–visual domain adaptation using conditional semi-supervised Generative Adversarial Networks

Athanasiadis, Christos; Hortal, Enrique; Asteriadis, Stylianos

doi:10.1016/j.neucom.2019.09.106

Cited by 29 publications

(27 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Common knowledge transfer in multi-modal methods include fine-tune well trained models to a specific type of signal (Vielzeuf et al, 2017;Yan et al, 2018;Huang et al, 2019;Ortega et al, 2019), or fine-tune different well-trained models to both speech and video signals (Ouyang et al, 2017;Zhang et al, 2017;Ma et al, 2019). Other usage of transfer learning for multi-modal methods includes leveraging the knowledge from one signal to another (e.g., video to speech) to reduce the potential bias (Athanasiadis et al, 2019). While these methods provide a promising performance, applying the more recent transfer learning methods, or utilizing more types of signals and leveraging knowledge between multiple signals, is likely to improve transfer learning and boost emotion recognition accuracy.…”

Section: Multi-modal Transfer Learning For Emotion Recognitionmentioning

confidence: 99%

A Review of Generalizable Transfer Learning in Automatic Emotion Recognition

Feng

Chaspari

2020

Front. Comput. Sci.

View full text Add to dashboard Cite

Automatic emotion recognition is the process of identifying human emotion from signals such as facial expression, speech, and text. Collecting and labeling such signals is often tedious and many times requires expert knowledge. An effective way to address challenges related to the scarcity of data and lack of human labels, is transfer learning. In this manuscript, we will describe fundamental concepts in the field of transfer learning and review work which has successfully applied transfer learning for automatic emotion recognition. We will finally discuss promising future research directions of transfer learning for improving the generalizability of automatic emotion recognition systems.

show abstract

Section: Multi-modal Transfer Learning For Emotion Recognitionmentioning

confidence: 99%

A Review of Generalizable Transfer Learning in Automatic Emotion Recognition

Feng

Chaspari

2020

Front. Comput. Sci.

View full text Add to dashboard Cite

show abstract

“…Similar tests with datasets CREMAD [21] & RVDSR [22] were done by researchers [23] with accuracy notes of 65%, 58.33%, [24] 52.52%, 47.11 %, [25] 74.0 %, 67.5 % and [26] 62.84% only for CREMAD [21] dataset.…”

Section: ( )mentioning

confidence: 55%

An Improvised Facial Emotion Recognition System using the Optimized Convolutional Neural Network Model with Dropout

Srinivas¹,

Mishra²

2021

IJACSA

View full text Add to dashboard Cite

Facial expression detection has long been regarded as both verbal and nonverbal communication. The muscular expression on a person's face reflects their physical and mental state. Using computer programming to integrate all face curves into a categorization class is significantly more important than doing so manually. Convolutional Neural Networks, an Artificial Intelligence approach, was recently developed to improve the task with more acceptance. Due to overfit during the learning step, the model performance may be lowered and regarded underperforming. There is a method dropout uses to reduce testing error. The influence of dropout is applied at convolutional layers and dense layers to classify face emotions into a distinct category of Happy, Angry, Sad, Surprise, Neutral, Disgust, and Fear and is represented as an improved convolutional neural network model. The experimental setup used the datasets namely JAFEE, CK48, FER2013, RVDSR, CREMAD and a selfprepared dataset of 36,153 facial images for observing train and test accuracy in presence and absence of dropout. Test accuracies of 92.33, 96.50, 97.78, 99.44, and 98.68 are obtained on Fer2013, RVDSR, CREMA-D, CK48, and JAFFE datasets are obtained in presence of dropout. The used features are countably large in the computation as a result the higher computation support of NVDIA with the capacity of GPU 16GB, CPU 13GB and memory 73.1 GB are used for the experimental purposes.

show abstract

“…Our work is closely related to the reconstruction from audio recordings. For example, Oh et al [11] and Athanasiadis et al [12] have used speech to reconstruct the speakers' face. Chen et al [13] Hao et al [14] use conditional GANs and cycle GAN to achieve the cross-modal audio-visual generation of musical performances.…”

Section: Audio-visual Cross-modal Learningmentioning

confidence: 99%

Cross-Modal Spectrum Transformation Network for Acoustic Scene Classification

Yang

Neophytou

Sengupta

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) with log-mel spectrum features have shown promising results for acoustic scene classification tasks. However, the performance of these CNN based classifiers is still lacking as they do not generalise well for unknown environments. To address this issue, we introduce an acoustic spectrum transformation network where traditional log-mel spectrums are transformed into imagined visual features (IVF). The imagined visual features are learned by exploiting the relationship between audio and visual features present in video recordings. An auto-encoder is used to encode images as visual features and a transformation network learns how to generate imagined visual features from log-mel. Our model is trained on a large dataset of Youtube videos. We test our proposed method on the scene classification task of DCASE and ESC-50, where our method outperforms other spectrum features, especially for unseen environments.

show abstract

Audio–visual domain adaptation using conditional semi-supervised Generative Adversarial Networks

Cited by 29 publications

References 25 publications

A Review of Generalizable Transfer Learning in Automatic Emotion Recognition

A Review of Generalizable Transfer Learning in Automatic Emotion Recognition

An Improvised Facial Emotion Recognition System using the Optimized Convolutional Neural Network Model with Dropout

Cross-Modal Spectrum Transformation Network for Acoustic Scene Classification

Contact Info

Product

Resources

About