StyleTalk: One-Shot Talking Head Generation with Controllable Speaking Styles

Ma, Yifeng; Wang, Shuzhen; Hu, Zhipeng; Cheng, Fan; Tangjie, Lv,; Ding, Yulong; Deng, Zhidong; Yu, Xin

doi:10.1609/aaai.v37i2.25280

Cited by 32 publications

(9 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Compared with audio-driven methods, video-driven methods can utilize richer information contained in the input video to generate more natural and realistic results, which can be roughly classified into 2D keypoint-based methods (Siarohin et al 2019;Zhao and Zhang 2022), 2D GANbased methods (Wang et al 2022;Yin et al 2022), and 3Dmodel-based networks (Lahiri et al 2021;Hong et al 2022;Ma et al 2023). 2D keypoint-based methods first compute the transfer matrix via matching keypoint pairs between the source image and the driving image, then wrap the source image to get the dense flow, and finally generate images with a GAN generator (Hong et al 2022).…”

Section: Related Workmentioning

confidence: 99%

FG-EmoTalk: Talking Head Video Generation with Fine-Grained Controllable Facial Expressions

Sun,

Xuan,

Liu

et al. 2024

AAAI

View full text Add to dashboard Cite

Although deep generative models have greatly improved one-shot video-driven talking head generation, few studies address fine-grained controllable facial expression editing, which is crucial for practical applications. Existing methods rely on a fixed set of predefined discrete emotion labels or simply copy expressions from input videos. This is limiting as expressions are complex, and methods using only emotion labels cannot generate fine-grained, accurate or mixed expressions. Generating talking head video with precise expressions is also difficult using 3D model-based approaches, as 3DMM only models facial movements and tends to produce deviations. In this paper, we propose a novel framework enabling fine-grained facial expression editing in talking face generation. Our goal is to achieve expression control by manipulating the intensities of individual facial Action Units (AUs) or groups. First, compared with existing methods which decouple the face into pose and expression, we propose a disentanglement scheme to isolates three components from the human face, namely, appearance, pose, and expression. Second, we propose to use input AUs to control muscle group intensities in the generated face, and integrate the AUs features with the disentangled expression latent code. Finally, we present a self-supervised training strategy with well-designed constraints. Experiments show our method achieves fine-grained expression control, produces high-quality talking head videos and outperforms baseline methods.

show abstract

Section: Related Workmentioning

confidence: 99%

FG-EmoTalk: Talking Head Video Generation with Fine-Grained Controllable Facial Expressions

Sun,

Xuan,

Liu

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Recent advancements (Ji et al 2021;Pan et al 2023;Tan, Ji, and Pan 2023) have also explored the synthesis of emotional expressions in talking faces. Ji et al (2021) and Sinha et al (2021) utilize one-hot emotion labels as input to generate emotional talking faces, while others (Ji et al 2022;Ma et al 2023b) resort to another video for emotion source. In contrast, our approach offers a more user-friendly control by allowing users to input easy-to-use text descriptions to suggest the desired emotion style.…”

Section: Related Workmentioning

confidence: 99%

“…Then, we pass the stylized images through several state-of-the-art (SOTA) talking face generation methods to achieve the same task as our proposed method. The comparing methods include MakeItTalk (Zhou et al 2020), Wav2Lip (Prajwal et al 2020), Audio2Head (Wang et al 2021), PC-AVS (Zhou et al 2021), AVCT (Wang et al 2022a), EAMM (Ji et al 2022) and StyleTalk (Ma et al 2023b), where only the latter two methods support talking head generation with emotion style. We assess the results using evaluation metrics including SSIM (Wang et al 2004), FID (Heusel et al 2017) and PSNR for image generation quality, M-LMD (Chen et al 2019) for accuracy evaluation of lip movement, F-LMD (Chen et al 2019) for emotion style evaluation.…”

Section: Experiments Experimental Settingsmentioning

confidence: 99%

“…Despite the progress achieved by AVCT, it neglects the emotion style. While EAMM and StyleTalk take an emotion source video for emotion style reference which provides more concrete style information than textual description used in our method(Ma et al 2023a), we perform more expressive emotion style than EAMM, which appears server facial deformation as circled in the red boxes. Talker) per iteration.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style

Tan,

Ji,

Pan

2024

AAAI

View full text Add to dashboard Cite

Although automatically animating audio-driven talking heads has recently received growing interest, previous efforts have mainly concentrated on achieving lip synchronization with the audio, neglecting two crucial elements for generating expressive videos: emotion style and art style. In this paper, we present an innovative audio-driven talking face generation method called Style2Talker. It involves two stylized stages, namely Style-E and Style-A, which integrate text-controlled emotion style and picture-controlled art style into the final output. In order to prepare the scarce emotional text descriptions corresponding to the videos, we propose a labor-free paradigm that employs large-scale pretrained models to automatically annotate emotional text labels for existing audio-visual datasets. Incorporating the synthetic emotion texts, the Style-E stage utilizes a large-scale CLIP model to extract emotion representations, which are combined with the audio, serving as the condition for an efficient latent diffusion model designed to produce emotional motion coefficients of a 3DMM model. Moving on to the Style-A stage, we develop a coefficient-driven motion generator and an art-specific style path embedded in the well-known StyleGAN. This allows us to synthesize high-resolution artistically stylized talking head videos using the generated emotional motion coefficients and an art style source picture. Moreover, to better preserve image details and avoid artifacts, we provide StyleGAN with the multi-scale content features extracted from the identity image and refine its intermediate feature maps by the designed content encoder and refinement network, respectively. Extensive experimental results demonstrate our method outperforms existing state-of-the-art methods in terms of audio-lip synchronization and performance of both emotion style and art style.

show abstract

“…3D Morphable Models (3DMM) is another effective intermediate representation. AudioDVP 43 predicts facial expression parameters of 3DMM from audio and then rerenders the reenacted face after replacing expression parameters computed from the original target video with the predicted one; StyleTalk 44 obtains the stylized expression parameters from audio and reference style video, then generates the output video using them and the identity reference image. In addition to the explicit motion representations, implicit features have also been studied extensively.…”

Section: Talking Head Generationmentioning

confidence: 99%

Wav2Lip‐HR: Synthesising clear high‐resolution talking head in the wild

Liang,

Wang,

Chen

et al. 2023

Computer Animation & Virtual

View full text Add to dashboard Cite

Talking head generation aims to synthesize a photo‐realistic speaking video with accurate lip motion. While this field has attracted more attention in recent audio‐visual researches, most existing methods do not achieve the simultaneous improvement of lip synchronization and visual quality. In this paper, we propose Wav2Lip‐HR, a neural‐based audio‐driven high‐resolution talking head generation method. With our technique, all required to generate a clear high‐resolution lip sync talking video is an image/video of the target face and an audio clip of any speech. The primary benefit of our method is that it generates clear high‐resolution videos with sufficient facial details, rather than the ones just be large‐sized with less clarity. We first analyze key factors that limit the clarity of generated videos and then put forth several important solutions to address the problem, including data augmentation, model structure improvement and a more effective loss function. Finally, we employ several efficient metrics to evaluate the clarity of images generated by our proposed approach as well as several widely used metrics to evaluate lip‐sync performance. Numerous experiments demonstrate that our method has superior performance on visual quality and lip synchronization when compared to other existing schemes.

show abstract

StyleTalk: One-Shot Talking Head Generation with Controllable Speaking Styles

Cited by 32 publications

References 42 publications

FG-EmoTalk: Talking Head Video Generation with Fine-Grained Controllable Facial Expressions

FG-EmoTalk: Talking Head Video Generation with Fine-Grained Controllable Facial Expressions

Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style

Wav2Lip‐HR: Synthesising clear high‐resolution talking head in the wild

Contact Info

Product

Resources

About