GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis

Ye, Zhenhui; Jiang, Ziyue Karen; Ren, Yang; Liu, Jinglin; He, JinZheng; Zhang, Zhao

doi:10.48550/arxiv.2301.13430

Cited by 5 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To tackle the above issues, there are methods to decouple the NeRF-based talking head generation process. Geneface [10] is the first method that attempts to achieve this process by facial landmarks. It utilizes variational auto-encoder (VAE) [31] to generate facial landmarks from audio, and then employs additional networks to refine these landmarks.…”

Section: B 3d-based Methodsmentioning

confidence: 99%

“…This insight sparks the idea of decoupling the NeRF-based talking head generation process through the utilization of facial landmarks. Actually, a few methods like [10], [11] have validated the potential of decoupling talking head generation via landmark-based neural radiation fields. However, they still have some limitations, such as the inability to generate landmarks that align with the training set distribution in a single attempt and the lack of precise control over the contribution of landmarks at each sampling point, which is also a common challenge faced by NeRF-based methods and leads to increased training time.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Decoupled Two-Stage Talking Head Generation via Gaussian-Landmark-Based Neural Radiance Fields

Ma,

Cao,

Zhang

2024

Preprint

View full text Add to dashboard Cite

Talking head generation based on neural radiance fields (NeRF) has gained prominence, primarily owing to its implicit 3D representation capabilities within neural networks. However, most NeRF-based methods often intertwine audio-tovideo conversion in a joint training process, resulting in challenges such as inadequate lip synchronization, limited learning efficiency, large memory requirement and lack of editability. In response to these issues, this paper introduces a fully decoupled NeRF-based method for generating talking head. This method separates the audio-to-video conversion into two stages through the use of facial landmarks. Notably, the Transformer network is used to establish the cross-modal connection between audio and landmarks effectively and generate landmarks conforming to the distribution of training data. Then, these landmarks are combined with Gaussian relative position coding to refine the sampling points on the rays, thereby constructing a dynamic neural radiation field conditioned on these landmarks for rendering the generated head. This decoupled setup enhances both the fidelity and flexibility of mapping audio to video with two independent small-scale networks. Additionally, it supports the generation of the torso part from the head-only image with deformable convolution, further enhancing the realism of the generated talking head. The experimental results demonstrate that our method excels in producing lifelike talking head, and the lightweight neural network models also exhibit superior speed and learning efficiency with less memory requirement.

show abstract

Section: B 3d-based Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Decoupled Two-Stage Talking Head Generation via Gaussian-Landmark-Based Neural Radiance Fields

Ma,

Cao,

Zhang

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…HuBERT Encoder To make full use of the information contained in the audio, we adopt a pre-trained HuBERT model to extract features. Instead of directly taking the final embedding as the subsequent input [36], we predict N hidden layers, which are weighted summed to feed into the Mon Decoder. The obtained HuBERT feature f h can be represented as:…”

Section: Architecturementioning

confidence: 99%

Expressive Talking Avatars

Pan,

Tan,

Cheng

et al. 2024

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Stylized avatars are common virtual representations used in VR to support interaction and communication between remote collaborators. However, explicit expressions are notoriously difficult to create, mainly because most current methods rely on geometric markers and features modeled for human faces, not stylized avatar faces. To cope with the challenge of emotional and expressive generating talking avatars, we build the Emotional Talking Avatar Dataset which is a talking-face video corpus featuring 6 different stylized characters talking with 7 different emotions. Together with the dataset, we also release an emotional talking avatar generation method which enables the manipulation of emotion. We validated the effectiveness of our dataset and our method in generating audio based puppetry examples, including comparisons to state-of-the-art techniques and a user study. Finally, various applications of this method are discussed in the context of animating avatars in VR.

show abstract

“…RAD-NeRF [12] decompose the inherently high-dimensional talking portrait representation into three low-dimensional feature grids, that makes can rending the talking portrait in real-time. GeneFace [14] propose a variational motion generator to generate accurate and expressive facial landmark and uses a NeRF-based renderer to render high-fidelity frames. Due to a lack of prior information, these tasks still struggle to render realistic expressions and natural movements.…”

Section: A Audio-driven Talking Face Generationmentioning

confidence: 99%