CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

Abstract

Large-scale pretrained transformers have created milestones in text (GPT-3) and text-to-image (DALL-E and CogView) generation. Its application to video generation is still facing many challenges: The potential huge computation cost makes the training from scratch unaffordable; The scarcity and weak relevance of text-video datasets hinder the model understanding complex movement semantics. In this work, we present 9B-parameter transformer CogVideo, trained by inheriting a pretrained text-to-image model, CogView2. We also propose multi-frame-rate hierarchical training strategy to better align text and video clips. As (probably) the first open-source large-scale pretrained text-to-video model, CogVideo outperforms all publicly available models at a large margin in machine and human evaluations. * Equal contribution. Preprint. Under review.

MusicLM: Generating Music From Text

“…Precursor to text-conditioned audio synthesis are the textconditioned image generation models, which made significant progress in quality due to architectural improvements and the availability of massive, high-quality paired training data. (Wu et al, 2022a;Hong et al, 2022;Villegas et al, 2022;Ho et al, 2022).…”

AVscript: Accessible Video Editing with Audio-Visual Scripts

“…For example, an editing system could describe the impact of an applied effect on the visual content in the video (e.g., "the vignette effect now covers the hands") using techniques from prior work in BLV visual design authoring [61] and computer vision approaches for captioning differences between pairs of similar images [32]. Recent strides in prompt-driven text generation [11], image generation [62,68], and image editing [52] suggest that prompt-driven video editing (e.g., make this clip moody) may be possible in the future [26]. Future research is needed to help BLV creators evaluate their results with such tools.…”

Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators

“…Phenaki [38] introduces a bidirectional masked transformer with a causal attention mechanism that allows the generation of arbitrary-long videos from text prompt sequences. CogVideo [15] extends the text-to-image model CogView 2 [4] by tuning it using a multi-frame-rate hierarchical training strategy to better align text and video clips. Video Diffusion Models (VDM) [14] naturally extend text-to-image diffusion models and train jointly on image and video data.…”