Diffusion Probabilistic Modeling for Video Generation

Yang, Ruihan; Srivastava, Prakhar; Mandt, Stephan

doi:10.48550/arxiv.2203.09481

Cited by 16 publications

(18 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Denoising diffusion models [60,64] have seen great success on a wide variety of different challenges, ranging from image2image translation tasks like inpainting, colorisation, image upscaling, uncropping [6,26,41,42,50,53,57,59], audio generation [11,28,33,35,38,48,67,80], text-based image generation [4,21,23,46,51,55,58], video generation [24,27,82,86], and many others. For a thorough review on diffusion models and all of their recent applications, we recommend [81].…”

Section: Diffusion Modelsmentioning

confidence: 99%

Speech Driven Video Editing via an Audio-Conditioned Diffusion Model

Bigioi¹,

Basak²,

Jordan³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this paper we propose a method for end-to-end speech driven video editing using a denoising diffusion model. Given a video of a person speaking, we aim to re-synchronise the lip and jaw motion of the person in response to a separate auditory speech recording without relying on intermediate structural representations such as facial landmarks or a 3D face model. We show this is possible by conditioning a denoising diffusion model with audio spectral features to generate synchronised facial motion. We achieve convincing results on the task of unstructured single-speaker video editing, achieving a word error rate of 45% using an off the shelf lip reading model. We further demonstrate how our approach can be extended to the multi-speaker domain. To our knowledge, this is the first work to explore the feasibility of applying denoising diffusion models to the task of audio-driven video editing. 1

show abstract

Section: Diffusion Modelsmentioning

confidence: 99%

Speech Driven Video Editing via an Audio-Conditioned Diffusion Model

Bigioi¹,

Basak²,

Jordan³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Diffusion Models for Image Synthesis. Starting with the seminal works of Sohl-Dickstein et al [52] and Ho et al [21], diffusion-based generative models have improved generative modeling of artificial visual systems [11,31,61,23,64,46] and other data [32,24,62] by sequentially removing noise from a random signal to generate an image. Being likelihood-based models, they achieve high data distribution coverage with well-behaved optimization properties while producing high resolution images at unprecedented quality.…”

Section: Related Workmentioning

confidence: 99%

Retrieval-Augmented Diffusion Models

Blattmann¹,

Rombach²,

Oktay³

et al. 2022

Preprint

View full text Add to dashboard Cite

Generative image synthesis with diffusion models has recently achieved excellent visual quality in several tasks such as text-based or class-conditional image synthesis. Much of this success is due to a dramatic increase in the computational capacity invested in training these models. This work presents an alternative approach: inspired by its successful application in natural language processing, we propose to complement the diffusion model with a retrieval-based approach and to introduce an explicit memory in the form of an external database. During training, our diffusion model is trained with similar visual features retrieved via CLIP and from the neighborhood of each training instance. By leveraging CLIP's joint image-text embedding space, our model achieves highly competitive performance on tasks for which it has not been explicitly trained, such as class-conditional or text-image synthesis, and can be conditioned on both text and image embeddings. Moreover, we can apply our approach to unconditional generation, where it achieves state-of-the-art performance. Our approach incurs low computational and memory overheads and is easy to implement. We discuss its relationship to concurrent work and will publish code and pretrained models soon.

show abstract

“…shape generation [4,51,85], video generation [26,83], Riemannian manifolds [16], symbolic music generation [54], guided image synthesis [52] and text-to-image [59]. In contrast to these works, our main focus is not to improve empirical results on existing diffusion models, analyse their behaviour or extend them to new data domains.…”

Section: Related Workmentioning

confidence: 99%

Generative Modelling With Inverse Heat Dissipation

Rissanen¹,

Heinonen²,

Solin³

2022

Preprint

View full text Add to dashboard Cite

While diffusion models have shown great success in image generation, their noiseinverting generative process does not explicitly consider the structure of images, such as their inherent multi-scale nature. Inspired by diffusion models and the desirability of coarse-to-fine modelling, we propose a new model that generates images through iteratively inverting the heat equation, a PDE that locally erases fine-scale information when run over the 2D plane of the image. In our novel methodology, the solution of the forward heat equation is interpreted as a variational approximation in a directed graphical model. We demonstrate promising image quality and point out emergent qualitative properties not seen in diffusion models, such as disentanglement of overall colour and shape in images and aspects of neural network interpretability. Spectral analysis on natural images positions our model as a type of dual to diffusion models and reveals implicit inductive biases in them.Preprint. Under review.

show abstract

Diffusion Probabilistic Modeling for Video Generation

Cited by 16 publications

References 0 publications

Speech Driven Video Editing via an Audio-Conditioned Diffusion Model

Speech Driven Video Editing via an Audio-Conditioned Diffusion Model

Retrieval-Augmented Diffusion Models

Generative Modelling With Inverse Heat Dissipation

Contact Info

Product

Resources

About