Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes

Bond-Taylor, Sam; Hessey, Peter; Sasaki, Hiroshi; Breckon, Toby P.; Willcocks, Chris G.

doi:10.1007/978-3-031-20050-2_11

Cited by 28 publications

(14 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some studies, such as SED ( Strudel et al, 2022 ), have pointed out the limitations of low sampling efficiency in diffusion models, which is indeed a drawback of diffusion models. To address this issue, in the field of computer vision, there have been a few studies that propose different efficient sampling strategies ( Bond-Taylor et al, 2022 ; Xiao, Kreis & Vahdat, 2022 ; Watson et al, 2022 ; Vahdat, Kreis & Kautz, 2021 ; Zhang & Chen, 2021 ). These methods have demonstrated the ability to double the sampling speed in many cases.…”

Section: Future Directionsmentioning

confidence: 99%

Diffusion models in text generation: a survey

Yi,

Chen,

Zhang

et al. 2024

PeerJ Computer Science

View full text Add to dashboard Cite

Diffusion models are a kind of math-based model that were first applied to image generation. Recently, they have drawn wide interest in natural language generation (NLG), a sub-field of natural language processing (NLP), due to their capability to generate varied and high-quality text outputs. In this article, we conduct a comprehensive survey on the application of diffusion models in text generation. We divide text generation into three parts (conditional, unconstrained, and multi-mode text generation, respectively) and provide a detailed introduction. In addition, considering that autoregressive-based pre-training models (PLMs) have recently dominated text generation, we conduct a detailed comparison between diffusion models and PLMs in multiple dimensions, highlighting their respective advantages and limitations. We believe that integrating PLMs into diffusion is a valuable research avenue. We also discuss current challenges faced by diffusion models in text generation and propose potential future research directions, such as improving sampling speed to address scalability issues and exploring multi-modal text generation. By providing a comprehensive analysis and outlook, this survey will serve as a valuable reference for researchers and practitioners interested in utilizing diffusion models for text generation tasks.

show abstract

Section: Future Directionsmentioning

confidence: 99%

Diffusion models in text generation: a survey

Yi,

Chen,

Zhang

et al. 2024

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…Diffusion models are widely used in Artificial Intelligence Generated Content (AIGC), which are of great importance in generative models. Diffusion models have illustrated their power in image generation [14,18,36,45], detection [7], segmentation [1,3,8], image-to-image translation [23,24], super resolution [38], image inpainting [5], image editing [31], text-to-image [2,15,25], video generation [17,40], point cloud [30,49,52], and human motion synthesis [22,39]. DDPM-Segmentation [3] is the first work to apply the diffusion model to semantic segmentation, which pre-trains a diffusion model and then trains classifiers for each pixel.…”

Section: Diffusion Modelmentioning

confidence: 99%

“…Diffusion models naturally have the ability to denoise noisy samples to the ideal data. Recently, the diffusion probability models (DPM) have illustrated their great power in generative tasks [5,22,25,31], but their potential in BEV perception tasks has not been fully explored. In this work, we propose DiffBEV, a novel framework that utilizes conditional DPM to improve quality of the BEV feature and push the boundary of BEV perception.…”

Section: Introductionmentioning

confidence: 99%

DiffBEV: Conditional Diffusion Model for Bird's Eye View Perception

Zou¹,

Zhu²,

Ye³

et al. 2023

Preprint

View full text Add to dashboard Cite

BEV perception is of great importance in the field of autonomous driving, serving as the cornerstone of planning, controlling, and motion prediction. The quality of the BEV feature highly affects the performance of BEV perception. However, taking the noises in camera parameters and Li-DAR scans into consideration, we usually obtain BEV representation with harmful noises. Diffusion models naturally have the ability to denoise noisy samples to the ideal data, which motivates us to utilize the diffusion model to get a better BEV representation. In this work, we propose an endto-end framework, named DiffBEV, to exploit the potential of diffusion model to generate a more comprehensive BEV representation. To the best of our knowledge, we are the first to apply diffusion model to BEV perception. In practice, we design three types of conditions to guide the training of the diffusion model which denoises the coarse samples and refines the semantic feature in a progressive way. What's more, a cross-attention module is leveraged to fuse the context of BEV feature and the semantic content of conditional diffusion model. DiffBEV achieves a 25.9% mIoU on the nuScenes dataset, which is 6.2% higher than the bestperforming existing approach. Quantitative and qualitative results on multiple benchmarks demonstrate the effectiveness of DiffBEV in BEV semantic segmentation and 3D object detection tasks. The code ‡ will be available soon.

show abstract

“…More recently, diffusion model-based approaches are gaining increasing popularity due to their exceptional results in image generation (Sohl-Dickstein et al, 2015;Ho et al, 2020;Yang et al, 2022;Bond-Taylor et al, 2021;Chung et al, 2022b;Batzolis et al, 2022;Bansal et al, 2022;Liu et al, 2022;Ku et al, 2022;Benton et al, 2022;Horwitz & Hoshen, 2022;Horita et al, 2022;Li et al, 2022). Besides, these methods enjoy the advantage of being able to perform inpainting without the need for degradation-specific training (Song & Ermon, 2019a).…”

Section: Related Workmentioning

confidence: 99%

MRM-Net: Towards Deep Multi-Resolution Mutual Learning for Progressive Image Inpainting

Zhang¹,

Zhang²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

<p>Deep image inpainting methods have improved the inpainting performance greatly due to the powerful representation ability of deep learning. However, current deep inpainting models still tend to produce unreasonable structures and blurry textures due to the ill-posed properties of the task, i.e., image inpainting is still a challenging topic. In this paper, we therefore propose a novel deep multi-resolution mutual learning (DMRML) strategy for progressive inpainting, which can fully explore the information from various resolutions. Specifically, we design a new image inpainting network, termed multi-resolution mutual network (MRM-Net), which takes the damaged images of various resolutions as input, then excavates and exploits the correlation among different resolutions to guide the image inpainting process. Technically, the setting of MRM-Net designs two new modules called multi-resolution information interaction (MRII) and adaptive content enhancement (ACE). MRII aims at discovering the correlation of multiple resolutions and exchanging information, and ACE focuses on enhancing the contents using the interacted features. We also present an memory preservation mechanism (MPM) to prevent from the information loss with the increasing layers. Extensive experiments on Paris Street View, Places2 and CelebA-HQ datasets demonstrate that our proposed MRM-Net can effectively recover the textures and structures, and performs favorably against other state-of-the-art methods. </p>

show abstract

Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes

Cited by 28 publications

References 41 publications

Diffusion models in text generation: a survey

Diffusion models in text generation: a survey

DiffBEV: Conditional Diffusion Model for Bird's Eye View Perception

MRM-Net: Towards Deep Multi-Resolution Mutual Learning for Progressive Image Inpainting

Contact Info

Product

Resources

About