EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model

Xinya, Ji,; Zhou, Hang; Wang, Kaisiyuan; Wu, Qianyi; Wu, Wayne; Xing, Feng; Cao, Xun

doi:10.48550/arxiv.2205.15278

Cited by 3 publications

(13 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We evaluate Diffused Heads on the most commonly used datasets for talking face generation: CREMA [2] and LRW [4]. We compare our method qualitatively and quantitatively to the current state-of-the-art in guided [15,24,45,46] and pose guidance-free [40] video synthesis. To experience the full quality of our results, readers are strongly encouraged to watch generated videos in the supplementary materials.…”

Section: Methodsmentioning

confidence: 99%

“…The paper [29] introduces a Portrait Image Neural Renderer (PIRenderer) that controls the face motions with the parameters of a three-dimensional morphable face model. In [15], Implicit Emotion Displacement Learner, together with Dense Warping Field, are used to obtain high-quality images.…”

Section: Related Workmentioning

confidence: 99%

“…Therefore, to make the model robust on sample initialization, we utilize x Id as a substitute for missing motion frames. The motion frames are added to Equation (15) and get the final form of the direct input to the model:…”

Section: Motion Framesmentioning

confidence: 99%

“…Unlike other methods [3,15,24,29,35,40,45], we do not use any explicit loss function to promote better lip sync of generated samples. Solutions that rely on using dedicated perceptual losses based on pre-trained synchronization or lipreading models have been effective in improving lip motion accuracy [24,39] but there are two problems prohibiting this approach with the proposed model.…”

Section: Lip Sync Lossmentioning

confidence: 99%

“…Generated heads move and behave in a natural expressive way while still preserving the subject's identity and plausible lip sync. In contrast to most recent approaches [3,15,24,29,35,40,[44][45][46], we use Denoising Diffusion Probabilistic Models [12,19] that utilize a variational approach instead of adversarial training and do not require stabilizing multiple discriminators. To eliminate the problem of unnaturally-looking sequences, we introduce motion frames (see Section 4.2) that are recurrently guiding video creation.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation

Stypułkowski¹,

Vougioukas²,

He³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Motion Framesmentioning

confidence: 99%

Section: Lip Sync Lossmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation

Stypułkowski¹,

Vougioukas²,

He³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

StyleTalk: One-Shot Talking Head Generation with Controllable Speaking Styles

Wang²,

Hu³

et al. 2023

AAAI

View full text Add to dashboard Cite

Different people speak with diverse personalized speaking styles. Although existing one-shot talking head methods have made significant progress in lip sync, natural facial expressions, and stable head motions, they still cannot generate diverse speaking styles in the final talking head videos. To tackle this problem, we propose a one-shot style-controllable talking face generation framework. In a nutshell, we aim to attain a speaking style from an arbitrary reference speaking video and then drive the one-shot portrait to speak with the reference speaking style and another piece of audio. Specifically, we first develop a style encoder to extract dynamic facial motion patterns of a style reference video and then encode them into a style code. Afterward, we introduce a style-controllable decoder to synthesize stylized facial animations from the speech content and style code. In order to integrate the reference speaking style into generated videos, we design a style-aware adaptive transformer, which enables the encoded style code to adjust the weights of the feed-forward layers accordingly. Thanks to the style-aware adaptation mechanism, the reference speaking style can be better embedded into synthesized videos during decoding. Extensive experiments demonstrate that our method is capable of generating talking head videos with diverse speaking styles from only one portrait image and an audio clip while achieving authentic visual effects. Project Page: https://github.com/FuxiVirtualHuman/styletalk.

show abstract

DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Video

Zhang

Deng

et al. 2023

AAAI

View full text Add to dashboard Cite

For few-shot learning, it is still a critical challenge to realize photo-realistic face visually dubbing on high-resolution videos. Previous works fail to generate high-fidelity dubbing results. To address the above problem, this paper proposes a Deformation Inpainting Network (DINet) for high-resolution face visually dubbing. Different from previous works relying on multiple up-sample layers to directly generate pixels from latent embeddings, DINet performs spatial deformation on feature maps of reference images to better preserve high-frequency textural details. Specifically, DINet consists of one deformation part and one inpainting part. In the first part, five reference facial images adaptively perform spatial deformation to create deformed feature maps encoding mouth shapes at each frame, in order to align with input driving audio and also the head poses of input source images. In the second part, to produce face visually dubbing, a feature decoder is responsible for adaptively incorporating mouth movements from the deformed feature maps and other attributes (i.e., head pose and upper facial expression) from the source feature maps together. Finally, DINet achieves face visually dubbing with rich textural details. We conduct qualitative and quantitative comparisons to validate our DINet on high-resolution videos. The experimental results show that our method outperforms state-of-the-art works.

show abstract

EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model

Cited by 3 publications

References 46 publications

Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation

Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation

StyleTalk: One-Shot Talking Head Generation with Controllable Speaking Styles

DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Video

Contact Info

Product

Resources

About