Vector-quantized Image Modeling with Improved VQGAN

Abstract

Pretraining language models with next-token prediction on massive text corpora has delivered phenomenal zero-shot, few-shot, transfer learning and multi-tasking capabilities on both generative and discriminative language tasks. Motivated by this success, we explore a Vector-quantized Image Modeling (VIM) approach that involves pretraining a Transformer to predict rasterized image tokens autoregressively. The discrete image tokens are encoded from a learned Vision-Transformerbased VQGAN (ViT-VQGAN). We first propose multiple improvements over vanilla VQGAN from architecture to codebook learning, yielding better efficiency and reconstruction fidelity. The improved ViT-VQGAN further improves vectorquantized image modeling tasks, including unconditional, class-conditioned image generation and unsupervised representation learning. When trained on Im-ageNet at 256 × 256 resolution, we achieve Inception Score (IS) of 175.1 and Fréchet Inception Distance (FID) of 4.17, a dramatic improvement over the vanilla VQGAN, which obtains 70.6 and 17.04 for IS and FID, respectively. Based on ViT-VQGAN and unsupervised pretraining, we further evaluate the pretrained Transformer by averaging intermediate features, similar to Image GPT (iGPT). This ImageNet-pretrained VIM-L significantly beats iGPT-L on linear-probe accuracy from 60.3% to 72.2% for a similar model size. ViM-L also outperforms iGPT-XL which is trained with extra web image data and larger model size.

MaskGIT: Masked Generative Image Transformer

“…VQGAN [15] adds adversarial loss and perceptual loss [26,54] in the first stage to improve the image fidelity. A contemporary work to ours, VIM [51], proposes to use a VIT backbone [13] to further improve the tokenization stage. Since these approaches still employ an auto-regressive model, the decoding time in the second stage scales with the token sequence length.…”

Corrupted Image Modeling for Self-Supervised Visual Pre-Training

“…In this paper, we directly use the same image tokenizer as BEiT for a fair and clear comparison with recent works. Meanwhile, we believe other image tokenizers, such as (Esser et al, 2021;Dong et al, 2021;Yu et al, 2021), deserve an indepth study for CIM pre-training in the future.…”

“…In this paper, we focus more on the architectural flexibility and universality of CIM, while the scaling behavior is not fully explored. The image tokenizer we use is essentially a large CNN and adds nontrivial overhead during pre-training, we believe that it can be largely resolved by using a more advanced tokenizer, such as ViT-VQGAN (Yu et al, 2021), which reports much higher throughput and better generation quality. Moreover, the influence of corrupted images' characteristics, styles and distributions on the pre-trained representation quality still needs more investigation.…”

“…For the first time, we demonstrate that both ViT and CNN can learn rich visual representations using a unified non-Siamese structure. Moreover, the components of CIM, such as the generator, the image tokenizer (Esser et al, 2021;Yu et al, 2021;Dong et al, 2021), the sampling method (Holtzman et al, 2019), as well as the pre-training objective (Wei et al, 2021) can be further customized and improved.…”

Multimodal Image Synthesis and Editing: A Survey

“…In above approaches, a convolution neural network (CNN) is learned to quantize and generate images. Instead, Yu et al [144] propose ViT-VQGAN which replaces the CNN encoder and decoder with Vision Transformer (ViT) [145]. Given sufficient data (for which unlabeled image data is plentiful), ViT-VQGAN is shown to be less constrained by the inductive priors imposed by convolutions.…”