Speech-Driven Facial Animation Using Polynomial Fusion of Features

Kefalas, Triantafyllos; Vougioukas, Konstantinos; Panagakis, Yannis; Petridis, Stavros; Kossaifi, Jean; Pantić, Maja

doi:10.1109/icassp40776.2020.9054469

Cited by 6 publications

(3 citation statements)

References 96 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More specifically, one needs to disentangle shape and motion factors and subsequently exploit them in reconstructing 3D visual objects. Face editing [178], [179] and talking faces synthesis [180] are further problems requiring the manipulation of multiple latent variables.…”

Section: A Multiple Factors Analysis In Computer Visionmentioning

confidence: 99%

Tensor Methods in Computer Vision and Deep Learning

et al. 2021

Self Cite

View full text Add to dashboard Cite

Tensors, or multidimensional arrays, are data structures that can naturally represent visual data of multiple dimensions. Inherently able to efficiently capture structured, latent semantic spaces and high-order interactions, tensors have a long history of applications in a wide span of computer vision problems. With the advent of the deep learning paradigm shift in computer vision, tensors have become even more fundamental. Indeed, essential ingredients in modern deep learning architectures, such as convolutions and attention mechanisms, can readily be considered as tensor mappings. In effect, tensor methods are increasingly finding significant applications in deep learning, including the design of memory and compute efficient network architectures, improving robustness to random noise and adversarial attacks, and aiding the theoretical understanding of deep networks.This article provides an in-depth and practical review of tensors and tensor methods in the context of representation learning and deep learning, with a particular focus on visual data analysis and computer vision applications. Concretely, besides fundamental work in tensor-based visual data analysis methods, we focus on recent developments that have brought on a gradual increase of tensor methods, especially in deep learning architectures, and their implications in computer vision applications. To further enable the newcomer to grasp such concepts quickly, we provide companion Python notebooks, covering key aspects of the paper and implementing them, step-by-step with TensorLy.

show abstract

Section: A Multiple Factors Analysis In Computer Visionmentioning

confidence: 99%

Tensor Methods in Computer Vision and Deep Learning

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…A VGG-M network pretrained on the VGG Face dataset [293] is employed in [281] to learn face features and a speech encoder modified from VGG-M is used to learn speech embedding. Three temporal encoders are used to extract representations of the speaker's identity, the audio segment, and the facial expressions, and a polynomial fusion layer is designed to generate a joint representation of the three encodings [260]. Differently, Mittal et al [266] develop a VAE to disentangle the phonetic content, emotional tone, and other factors into different representations solely from the input audio signal.…”

Section: Multi-modal Feature Extractionmentioning

confidence: 99%

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Liu

Wang

et al. 2021

Front. Neurorobot.

View full text Add to dashboard Cite

The redundant information, noise data generated in the process of single-modal feature extraction, and traditional learning algorithms are difficult to obtain ideal recognition performance. A multi-modal fusion emotion recognition method for speech expressions based on deep learning is proposed. Firstly, the corresponding feature extraction methods are set up for different single modalities. Among them, the voice uses the convolutional neural network-long and short term memory (CNN-LSTM) network, and the facial expression in the video uses the Inception-Res Net-v2 network to extract the feature data. Then, long and short term memory (LSTM) is used to capture the correlation between different modalities and within the modalities. After the feature selection process of the chi-square test, the single modalities are spliced to obtain a unified fusion feature. Finally, the fusion data features output by LSTM are used as the input of the classifier LIBSVM to realize the final emotion recognition. The experimental results show that the recognition accuracy of the proposed method on the MOSI and MELD datasets are 87.56 and 90.06%, respectively, which are better than other comparison methods. It has laid a certain theoretical foundation for the application of multimodal fusion in emotion recognition.

show abstract

“…The input of the decoder is the output of the encoder adding the 10 dimensional random noise. As described in [27], the random noise can introduce some natural variability during the audio-visual generation task. We used multiple random noises to generate various kinds of mapping gestures from one speech audio.…”

Section: B Gan Model For Speech-driven Gesture Generationmentioning

confidence: 99%

SRG³: Speech-driven Robot Gesture Generation with GAN

Tapus

2020

2020 16th International Conference on Control, Automation, Robotics and Vision (ICARCV)

View full text Add to dashboard Cite

Fig. 1: Our robot gesture generation pipeline. The speech audio is used as an input to the 3D pose synthesizer with a random noise. Then, the natural and human-like 3D gesture sequences (in the joint position(x, y, z) space) are generated. Moreover, one same speech audio with multiple different random noises can align multiple natural gesture expressions in the same manner as humans have similar but different gestures while expressing the same speech in different contexts and situations. The gesture retargeting part maps the obtained gestures in the joint position space of the Pepper robot (i.e., in the joint angle(pitch, roll, yaw) space). Finally, the mapped positions are applied on the real Pepper robot in the human-robot interaction.

show abstract

Speech-Driven Facial Animation Using Polynomial Fusion of Features

Cited by 6 publications

References 96 publications

Tensor Methods in Computer Vision and Deep Learning

Tensor Methods in Computer Vision and Deep Learning

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

SRG³: Speech-driven Robot Gesture Generation with GAN

Contact Info

Product

Resources

About

Speech-Driven Facial Animation Using Polynomial Fusion of Features

Cited by 6 publications

References 96 publications

Tensor Methods in Computer Vision and Deep Learning

Tensor Methods in Computer Vision and Deep Learning

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

SRG3: Speech-driven Robot Gesture Generation with GAN

Contact Info

Product

Resources

About

SRG³: Speech-driven Robot Gesture Generation with GAN