CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers

Abstract

The development of the transformer-based text-to-image models are impeded by its slow generation and complexity for high-resolution images. In this work, we put forward a solution based on hierarchical transformers and local parallel autoregressive generation. We pretrain a 6B-parameter transformer with a simple and flexible self-supervised task, Cross-modal general language model (CogLM), and finetune it for fast super-resolution. The new text-to-image system, CogView2, shows very competitive generation compared to concurrent state-of-the-art DALL-E-2, and naturally supports interactive text-guided editing on images.A lion man is typing in the office.A beautiful girl is hugging a husky. A lion teacher wearing a suit is in front of a blackboard.A robot is riding under the blue and cloudy sky.

CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

“…In the text-to-image generation, pretrained autoregressive transformers such as DALL-E [18] and CogView [5] have shown superiority in open-domain image generation. Besides the pure GPT-style generation, CogView2 [6] proposes a new language model CogLM for infilling in the image generation.…”

“…We train another frame interpolation model to insert transition frames to the generated samples of the sequential generation model. Thanks to the generality of CogLM [6], the two models can share the same structure and training process only with different attention masks.…”

“…However, most previous works use GPT [34,36,35], which is unidirectional. To be aware of the bidirectional context, we adopt Cross-Modal General Language Model (CogLM) proposed in [6] which unites bidirectional context-aware mask prediction and autoregressive generation by dividing tokens into unidirectional and bidirectional attention regions. While bidirectional regions can attend to all bidirectional regions, unidirectional regions can attend to all bidirectional regions and previous unidirectional regions.…”

“…Pretrained text-to-image models, e.g. CogView2 [6], already have a good command of the textimage relations. The coverage of the dataset to train these models is also larger than that of videos.…”

“…Here we present a large-scale pretrained text-to-video generative model, CogVideo, which is of 9.4 billion parameters and trained on 5.4 million text-video pairs. We build CogVideo based on a pretrained text-to-image model, CogView2 [6], in order to inherit the knowledge learned from the text-image pretraining. To ensure the alignment between text and its temporal counterparts in the video, we propose the multi-frame-rate hierarchical training.…”