MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation

Abstract

Video prediction is a challenging task. The quality of video frames from current state-of-the-art (SOTA) generative models tends to be poor and generalization beyond the training data is difficult. Furthermore, existing prediction frameworks are typically not capable of simultaneously handling other video-related tasks such as unconditional generation or interpolation. In this work, we devise a generalpurpose framework called Masked Conditional Video Diffusion (MCVD) for all of these video synthesis tasks using a probabilistic conditional score-based denoising diffusion model, conditioned on past and/or future frames. We train the model in a manner where we randomly and independently mask all the past frames or all the future frames. This novel but straightforward setup allows us to train a single model that is capable of executing a broad range of video tasks, specifically: future/past prediction -when only future/past frames are masked; unconditional generation -when both past and future frames are masked; and interpolation -when neither past nor future frames are masked. Our experiments show that this approach can generate high-quality frames for diverse types of videos. Our MCVD models are built from simple non-recurrent 2D-convolutional architectures, conditioning on blocks of frames and generating blocks of frames. We generate videos of arbitrary lengths autoregressively in a block-wise manner. Our approach yields SOTA results across standard video prediction and interpolation benchmarks, with computation times for training models measured in 1-12 days using ≤ 4 GPUs. Project page: https://mask-cond-video-diffusion.github.io Code: https://github.com/voletiv/mcvd-pytorch * Equal Contribution Preprint. Under review.

Predicting Player Engagement in Tom Clancy's The Division 2: A Multimodal Approach via Pixels and Gamepad Actions

“…All Layers (𝑀 𝑆𝑆𝐴𝐿 ). Following [52], we explore the case of time-conditioning each layer as long as it is onedimensional: e.g. in the frame model (Fig.…”

MaskViT: Masked Visual Pre-Training for Video Prediction

“…The video prediction task refers to the problem of generating videos conditioned on past frames [18,19], possibly with an additional natural language description [20-22, 22, 23] and/or motor commands [24][25][26][27]. Multiple classes of generative models have been utilized to tackle this problem, such as Generative adversarial networks (GANs) [28][29][30], Variational Autoencoders (VAEs) [31,32,[25][26][27], VQ-VAEs [33,34] and diffusion models [35,36]. Our work focuses on predicting future frames conditioned on past frames or motor commands and belongs to the family of two-stage methods that first encode the videos into a downsampled latent space and then use transformers to model an autoregressive prior [33,34].…”

Continuous conditional video synthesis by neural processes

“…Almost all the VFP models are autoregressive models based on ConvLSTMs or Transformers [35,36]. Recently, a few promising nonautoregressive VFP models were proposed [37,38,39]. By combining ConvLSTMs with an neural ordinary differential equation (ODE) solver, Vid-ODE [16] is the first method that unifies the VFP and VFI into a single model, and it is able to generate temporally continuous video.…”

“…Another work, masked conditional video diffusion (MCVD) [39] extends the 3D CNN-based diffusion models for video generation, but it is not an NP model and it is not able to do continuous video synthesis. In contrast, benefiting from the flexibility of NPs, our model is able to perform video random missing frames completion (VRC), which is more flexible than MCVD.…”