Fine-Grained Semantic Image Synthesis with Object-Attention Generative Adversarial Network

Abstract

Semantic image synthesis is a new rising and challenging vision problem accompanied by the recent promising advances in generative adversarial networks. The existing semantic image synthesis methods only consider the global information provided by the semantic segmentation mask, such as class label, global layout, and location, so the generative models cannot capture the rich local fine-grained information of the images (e.g., object structure, contour, and texture). To address this issue, we adopt a multi-scale feature fusion algorithm to refine the generated images by learning the fine-grained information of the local objects. We propose OA-GAN, a novel object-attention generative adversarial network that allows attention-driven, multi-fusion refinement for fine-grained semantic image synthesis. Specifically, the proposed model first generates multi-scale global image features and local object features, respectively, then the local object features are fused into the global image features to improve the correlation between the local and the global. In the process of feature fusion, the global image features and the local object features are fused through the channel-spatial-wise fusion block to learn ‘what’ and ‘where’ to attend in the channel and spatial axes, respectively. The fused features are used to construct correlation filters to obtain feature response maps to determine the locations, contours, and textures of the objects. Extensive quantitative and qualitative experiments on COCO-Stuff, ADE20K and Cityscapes datasets demonstrate that our OA-GAN significantly outperforms the state-of-the-art methods.

Lightweight semantic image synthesis with mutable framework

“…28, and the results on ADE20K and Cityscapes reported by Ref. 27. The sixth and seventh results are reported by Ref.…”

“…Liu et al 26 proposed to predict convolutional kernels conditioned on the semantic label map to generate the intermediate feature maps from the noise maps, which helps better exploit the semantic layout. Wang et al 27 proposed a method that allows attention-driven, multi-fusion refinement for fine-grained semantic image synthesis. Chen et al 28 presented a model that can be trained on only single or multiple pairs of images and semantic maps.…”

Semantic Image Synthesis via Diffusion Models

“…Semantic image synthesis [3,34,50,33,24,22,12,46,47,48,59,43,58,14,52,23,49,26,45,28] transforms semantic layouts into diverse realistic images. Recent work on semantic image synthesis is GAN-based and trained with the adversarial loss along with the reconstruction loss.…”