MaskGIT: Masked Generative Image Transformer

Chang, Hui‐Wen; Zhang, Han; J, Lu; Liu, Ce; Freeman, William T.

doi:10.48550/arxiv.2202.04200

Cited by 12 publications

(24 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bidirectional transformer for autoregressive generation is recently proposed by MaskGIT [4]. Here we show our model can also be combined with bidirectional transformer.…”

Section: Bidirectional Transformermentioning

confidence: 87%

See 1 more Smart Citation

3DILG: Irregular Latent Grids for 3D Generative Modeling

Zhang¹,

Nießner²,

Wonka³

2022

Preprint

View full text Add to dashboard Cite

We propose a new representation for encoding 3D shapes as neural fields. The representation is designed to be compatible with the transformer architecture and to benefit both shape reconstruction and shape generation. Existing works on neural fields are grid-based representations with latents defined on a regular grid. In contrast, we define latents on irregular grids, enabling our representation to be sparse and adaptive. In the context of shape reconstruction from point clouds, our shape representation built on irregular grids improves upon grid-based methods in terms of reconstruction accuracy. For shape generation, our representation promotes high-quality shape generation using auto-regressive probabilistic models. We show different applications that improve over the current state of the art. First, we show results for probabilistic shape reconstruction from a single higher resolution image. Second, we train a probabilistic model conditioned on very low resolution images. Third, we apply our model to category-conditioned generation. All probabilistic experiments confirm that we are able to generate detailed and high quality shapes to yield the new state of the art in generative 3D shape modeling.Preprint. Under review.

show abstract

“…Bidirectional transformer for autoregressive generation is recently proposed by MaskGIT [4]. Here we show our model can also be combined with bidirectional transformer.…”

Section: Bidirectional Transformermentioning

confidence: 87%

“…Tokens are sampled based on their probabilities (transformer output). See [4] for a detailed explanation. We show visualization of decoding steps in Fig.…”

Section: Bidirectional Transformermentioning

confidence: 99%

3DILG: Irregular Latent Grids for 3D Generative Modeling

Zhang¹,

Nießner²,

Wonka³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The speed of generation was not an issue at the era when GANs dominated the image generation, while constitutes a considerate challenge for current auto-regressive text-to-image models. M6-UFC [33] first introduces NAR methods into the VQ-VAE framework, and similar ideas are adopted by VQ-diffusion [11] and MaskGIT [1]. A possible drawback of pure NAR methods is that tokens sampled at the meantime might lead to global inconsistency in later steps during the generation of complex scenes.…”

Section: Related Workmentioning

confidence: 99%

CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers

Ding¹,

Zheng²,

Hong³

et al. 2022

Preprint

View full text Add to dashboard Cite

The development of the transformer-based text-to-image models are impeded by its slow generation and complexity for high-resolution images. In this work, we put forward a solution based on hierarchical transformers and local parallel autoregressive generation. We pretrain a 6B-parameter transformer with a simple and flexible self-supervised task, Cross-modal general language model (CogLM), and finetune it for fast super-resolution. The new text-to-image system, CogView2, shows very competitive generation compared to concurrent state-of-the-art DALL-E-2, and naturally supports interactive text-guided editing on images.A lion man is typing in the office.A beautiful girl is hugging a husky. A lion teacher wearing a suit is in front of a blackboard.A robot is riding under the blue and cloudy sky.

show abstract

“…The goal of MJP is to enhance the capacity of position insensitivity that directly increases the difficulties of image recovery from gradient updates, while to preserve the accuracy on the standard classification in the pre-training. In practice, image masking methods are commonly applied in recent vision task [2,57,20,58,7], which is a useful and off-the-shelf strategy for self-supervised image reconstruction. Given an input image x ∈ R H×W ×C , we reshape it into a sequence of flattened 2D patches x p ∈ R N ×(P 2 •C) , where (H, W ) is the resolution of the original image, C is the number of channels, (P, P ) is the resolution of each image patch, and N = HW/P 2 is the resulting number of patches.…”

Section: Masked Jigsaw Puzzlementioning

confidence: 99%

Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformers

Liu¹,

Ren²,

Song³

et al. 2022

Preprint

View full text Add to dashboard Cite

User privacy is of great concern in Federated Learning, while Vision Transformers (ViTs) have been revealed to be vulnerable to gradient-based inversion attacks. We show that the learned low-dimensional spatial prior in position embeddings (PEs) accelerates the training of ViTs. As a side effect, it makes the ViTs tend to be position sensitive and at high risk of privacy leakage. We observe that enhancing the position-insensitive property of a ViT model is a promising way to protect data privacy against these gradient attacks. However, simply removing the PEs may not only harm the convergence and accuracy of ViTs but also places the model at more severe privacy risk. To deal with the aforementioned contradiction, we propose a simple yet efficient Masked Jigsaw Puzzle (MJP) method to break the chain of gradient leakage in ViTs. MJP can be easily plugged into existing ViTs and their derived variants. Extensive experiments demonstrate that our proposed MJP method not only boosts the performance on large-scale datasets (i.e., ImageNet-1K), but can also improve the privacy preservation capacity in the typical gradient attacks by a large margin. Our code is available at: https://github.com/yhlleo/MJP.

show abstract

MaskGIT: Masked Generative Image Transformer

Cited by 12 publications

References 18 publications

3DILG: Irregular Latent Grids for 3D Generative Modeling

3DILG: Irregular Latent Grids for 3D Generative Modeling

CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers

Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformers

Contact Info

Product

Resources

About