Diverse Semantic Image Synthesis via Probability Distribution Modeling

Tan, Zhentao; Chai, Menglei; Chen, Dongdong; Liao, Jing; Chu, Qi; Liu, Bin; Hua, Gang; Yu, Nenghai

doi:10.1109/cvpr46437.2021.00787

Cited by 56 publications

(24 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Baselines. We compare our method with several semantic image editing baselines: SESAME [Ntavelis et al 2020], INADE [Tan et al 2021], Taming Transformers (TT) [Esser et al 2021a], and ImageBART [Esser et al 2021b]. SESAME and INADE are based on convolutional networks and only support image resolutions of up to 256 × 256 pixels.…”

Section: Resultsmentioning

confidence: 99%

ASSET: Autoregressive Semantic Scene Editing with Transformers at High Resolutions

Liu¹,

Shetty²,

Hinz³

et al. 2022

Preprint

View full text Add to dashboard Cite

and context, leading to synthesizing interesting phenomena in scenes, such as reflections of landscapes onto water or flora consistent with the rest of the landscape, that were not possible to generate reliably with previous convnets and transformer approaches. We present qualitative and quantitative results, along with user studies, demonstrating the effectiveness of our method. Our code and dataset are available at our project page: https://github.com/DifanLiu/ASSET CCS Concepts: • Computing methodologies → Image processing; Neural networks; Image representations.

show abstract

Section: Resultsmentioning

confidence: 99%

ASSET: Autoregressive Semantic Scene Editing with Transformers at High Resolutions

Liu¹,

Shetty²,

Hinz³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The aforementioned methods mainly focus on generating real and semantically-corresponding unimodal result. Paralleled with these methods, some other methods [57,12,59,44] explore multimodal generation, which is also a core target for one-to-many problems like semantic image synthesis. To tackle this issue, BicycleGAN [57] encourages bidirectional mapping between the generated image and latent code, and DSCGAN [12] propose a simple regularization loss to penalize the generator from mode collapse.…”

Section: Semantic Image Synthesismentioning

confidence: 99%

“…To tackle this issue, BicycleGAN [57] encourages bidirectional mapping between the generated image and latent code, and DSCGAN [12] propose a simple regularization loss to penalize the generator from mode collapse. More recently, INADE [44] proposes a framework that supports diverse generation at the instance level by instanceadaptive stochastic sampling. However, these multimodal methods still fail to obtain satisfactory results on generation quality and learned correspondence.…”

Section: Semantic Image Synthesismentioning

confidence: 99%

Semantic Image Synthesis via Diffusion Models

Wang¹,

Bao²,

Zhou³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

“…Current conditional generative models can synthesize images according to additional conditions such as image (Choi et al 2020;Lee et al 2020;Liu et al 2021), label (Chen et al 2019;Yang et al 2021), segmentation mask (Park et al 2019;Huang et al 2020;Tan et al 2021), text (Xu et al 2018;Qiao et al 2019;Li et al 2019a;Zhu et al 2019;Li et al 2019b) and layout (Sun and Wu 2019; Ashual and Wolf 2019;Zhao et al 2019). The text conditions can either be natural language sentences or scene graphs (Li et al 2019c;Tseng et al 2020).…”

Section: Related Work Conditional Image Generationmentioning

confidence: 99%

Hierarchical Image Generation via Transformer-Based Sequential Patch Selection

2022

AAAI

View full text Add to dashboard Cite

To synthesize images with preferred objects and interactions, a controllable way is to generate the image from a scene graph and a large pool of object crops, where the spatial arrangements of the objects in the image are defined by the scene graph while their appearances are determined by the retrieved crops from the pool. In this paper, we propose a novel framework with such a semi-parametric generation strategy. First, to encourage the retrieval of mutually compatible crops, we design a sequential selection strategy where the crop selection for each object is determined by the contents and locations of all object crops that have been chosen previously. Such process is implemented via a transformer trained with contrastive losses. Second, to generate the final image, our hierarchical generation strategy leverages hierarchical gated convolutions which are employed to synthesize areas not covered by any image crops, and a patch guided spatially adaptive normalization module which is proposed to guarantee the final generated images complying with the crop appearance and the scene graph. Evaluated on the challenging Visual Genome and COCO-Stuff dataset, our experimental results demonstrate the superiority of our proposed method over existing state-of-the-art methods.

show abstract

Diverse Semantic Image Synthesis via Probability Distribution Modeling

Cited by 56 publications

References 22 publications

ASSET: Autoregressive Semantic Scene Editing with Transformers at High Resolutions

ASSET: Autoregressive Semantic Scene Editing with Transformers at High Resolutions

Semantic Image Synthesis via Diffusion Models

Hierarchical Image Generation via Transformer-Based Sequential Patch Selection

Contact Info

Product

Resources

About