AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis

Guo, Yudong; Chen, Keyu; Liang, Shuhua; Liu, Yong-Jin; Bao, Hujun; Zhang, Jie

doi:10.1109/iccv48922.2021.00573

Cited by 250 publications

(121 citation statements)

References 44 publications

Supporting

Mentioning

119

Contrasting

Unclassified

Order By: Relevance

“…Guo et al [21] use an off-the-shelf parsing method [25] to divide training images into head and torso for individual NeRF modeling. Following their settings, we assume that the semantic parsing maps are also available in our method.…”

Section: Preliminaries and Problem Settingmentioning

confidence: 99%

“…An off-the-shelf method [25] is leveraged to obtain parsing maps of total 11 semantic classes. The background image and head pose are estimated in a similar way to Guo et al [21]. Note that the estimated head pose p h (t) at time t is treated as camera pose and the canonical pose p c is set as the starting frame's head pose, i.e., p c = p h (0) in Eq.…”

Section: Dataset and Preprocessingmentioning

confidence: 99%

“…The reconstruction/model-based methods require large-corpus training data or long videos, hence we directly inference with their publicly-released best models. Note that all baseline methods except for [21] fail to generate the whole portrait with full resolution, we divide our comparisons into two settings: 1) The cropped setting in Table 1, where we crop the generated facial image with same region and resize into same size for fair evaluation metric comparison.…”

Section: Quantitative Evaluationmentioning

confidence: 99%

“…2) The full resolution setting in Table 2, where we compare with AD-NeRF [21] that could also synthesize the whole portrait with full resolution of 450 × 450.…”

Section: Quantitative Evaluationmentioning

confidence: 99%

“…The rating is of scale 1-5, with the larger the better. We compare the lip-sync accuracy, video realness and image quality of baseline methods [9,21,47,59,61,83,84] and our proposed SSP-NeRF.…”

Section: Quantitative Evaluationmentioning

confidence: 99%

See 4 more Smart Citations

Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation

Liu¹,

Xu²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Animating high-fidelity video portrait with speech audio is crucial for virtual reality and digital entertainment. While most previous studies rely on accurate explicit structural information, recent works explore the implicit scene representation of Neural Radiance Fields (NeRF) for realistic generation. In order to capture the inconsistent motions as well as the semantic difference between human head and torso, some work models them via two individual sets of NeRF, leading to unnatural results. In this work, we propose Semantic-aware Speaking Portrait NeRF (SSP-NeRF), which creates delicate audio-driven portraits using one unified set of NeRF. The proposed model can handle the detailed local facial semantics and the global head-torso relationship through two semantic-aware modules. Specifically, we first propose a Semantic-Aware Dynamic Ray Sampling module with an additional parsing branch that facilitates audio-driven volume rendering. Moreover, to enable portrait rendering in one unified neural radiance field, a Torso Deformation module is designed to stabilize the large-scale non-rigid torso motions. Extensive evaluations demonstrate that our proposed approach renders more realistic video portraits compared to previous methods. Project page: https://alvinliu0.github.io/projects/SSP-NeRF.

show abstract

Section: Preliminaries and Problem Settingmentioning

confidence: 99%

Section: Dataset and Preprocessingmentioning

confidence: 99%

Section: Quantitative Evaluationmentioning

confidence: 99%

“…2) The full resolution setting in Table 2, where we compare with AD-NeRF [21] that could also synthesize the whole portrait with full resolution of 450 × 450.…”

Section: Quantitative Evaluationmentioning

confidence: 99%

Section: Quantitative Evaluationmentioning

confidence: 99%

See 3 more Smart Citations

Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation

Liu¹,

Xu²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Disentangling audio content and emotion with adaptive instance normalization for expressive facial animation synthesis

Chang

Zhao

Zhang

et al. 2022

Computer Animation & Virtual

View full text Add to dashboard Cite

3D facial animation synthesis from audio has been a focus in recent years. However, most existing literature works are designed to map audio and visual content, providing limited knowledge regarding the relationship between emotion in audio and expressive facial animation. This work generates audio-matching facial animations with the specified emotion label. In such a task, we argue that separating the content from audio is indispensable-the proposed model must learn to generate facial content from audio content while expressions from the specified emotion. We achieve it by an adaptive instance normalization module that isolates the content in the audio and combines the emotion embedding from the specified label. The joint content-emotion embedding is then used to generate 3D facial vertices and texture maps. We compare our method with state-of-the-art baselines, including the facial segmentation-based and voice conversion-based disentanglement approaches. We also conduct a user study to evaluate the performance of emotion conditioning. The results indicate that our proposed method outperforms the baselines in animation quality and expression categorization accuracy.

show abstract

Wav2Lip‐HR: Synthesising clear high‐resolution talking head in the wild

Liang,

Wang,

Chen

et al. 2023

Computer Animation & Virtual

View full text Add to dashboard Cite

Talking head generation aims to synthesize a photo‐realistic speaking video with accurate lip motion. While this field has attracted more attention in recent audio‐visual researches, most existing methods do not achieve the simultaneous improvement of lip synchronization and visual quality. In this paper, we propose Wav2Lip‐HR, a neural‐based audio‐driven high‐resolution talking head generation method. With our technique, all required to generate a clear high‐resolution lip sync talking video is an image/video of the target face and an audio clip of any speech. The primary benefit of our method is that it generates clear high‐resolution videos with sufficient facial details, rather than the ones just be large‐sized with less clarity. We first analyze key factors that limit the clarity of generated videos and then put forth several important solutions to address the problem, including data augmentation, model structure improvement and a more effective loss function. Finally, we employ several efficient metrics to evaluate the clarity of images generated by our proposed approach as well as several widely used metrics to evaluate lip‐sync performance. Numerous experiments demonstrate that our method has superior performance on visual quality and lip synchronization when compared to other existing schemes.

show abstract

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis

Cited by 250 publications

References 44 publications

Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation

Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation

Disentangling audio content and emotion with adaptive instance normalization for expressive facial animation synthesis

Wav2Lip‐HR: Synthesising clear high‐resolution talking head in the wild

Contact Info

Product

Resources

About