VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild

Abstract

We present VideoReTalking, a new system to edit the faces of a real-world talking head video according to input audio, producing a high-quality and lipsyncing output video even with a different emotion. Our system disentangles this objective into three sequential tasks: (1) face video generation with a canonical expression; (2) audio-driven lip-sync; and (3) face enhancement for improving photo-realism. Given a talking-head video, we first modify the expression of each frame according to the same expression template

STRDD: Scene Text Removal with Diffusion Probabilistic Models

“…Text-to-Video diffusion models [1,2,3,4,5,6,7,8,9,10,11] have become a powerful tool for producing high-quality video content from textual prompts. Pika [9] is a commercial text-to-video model by Pika Labs, which advances the field of video generation.…”

“…(2) We provide a detailed and in-depth comparison with the text-to-image prompt-gallery dataset, DiffusionDB, and highlight the necessity of VidProM as well as real users' preference. (3) We reveal several exciting research directions inspired by VidProM and position it as a rich database for future studies.…”

“…This breakthrough provides new possibilities for storytelling, immersive experiences, and content creation, as Sora [6] transforms textual descriptions into high-quality videos with ease. However, Sora [6] and other text-to-video diffusion models [9, 1,3,5] heavily relies on the prompts used. Despite their importance, there is no publicly available dataset focusing on text-to-video prompts, which hinders the development and evaluation of these models.…”

“…Our large-scale VidProM includes 1.67 million unique text-to-video prompts from real users and 6.69 million generated videos by 4 state-of-the-art diffusion models. The prompts are from official Pika Discord channels, and the videos are generated by Pika [9], Text2Video-Zero [1], VideoCraft2 [3], and ModelScope [5]. We distribute the generation process across 10 servers, each equipped with 8 Nvidia V100 GPUs.…”

Epidemiological Characteristics of Rubella in Beijing Haidian District of China, 2005 - 2020

“…Beyond faking faces or voices related to a person's identity through audio deepfake [23,36] or visual deepfake [25,41,44] techniques within each modality, malicious attackers can combine these technologies to create multimodal forged content, where both audio and visuals can be fake. Moreover, cutting-edge deepfake methods like Wav2Lip [37]and VideoReTalking [10] can even achieve crossmodal forgery by driving audio to generate precise lip-sync videos. Consequently, it is evident that deepfakes now encompass audio, visual, and even cross-modal forms.…”