Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creation

Akimoto, Naofumi; Matsuo, Yuhi; Aoki, Yoshimitsu

doi:10.1109/cvpr52688.2022.01115

Cited by 18 publications

(5 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Spherical structure and texture are modeled using cube maps (Han and Suh 2020), cylinder convolution (Liao et al 2022), predicted panoramic three-dimensional structures (Song et al 2018), and scene symmetry (Hara, Mukuta, and Harada 2021).Most previous generative models are limited in handling fixed scenes and low-resolution images. Recent research (Sumantri and Park 2020;Akimoto, Matsuo, and Aoki 2022a) addresses these limitations using hierarchical synthesis networks (Sumantri and Park 2020), U-Net structures (Akimoto, Matsuo, and Aoki 2022a), and separate models for generating and upscaling low-resolution images (Chen, Wang, and Liu 2022). Our proposed method tackles high-resolution panorama context and preserves spherical structure within a single module.…”

Section: Related Workmentioning

confidence: 99%

“…However, simply adopting current state-of-the-art methods such as DALL•E-like (Esser, Rombach, and Ommer 2021) for high-resolution panorama generation can result in spherical distortion and low efficiency. Recent efforts have attempted to address this issue by using separate models to first generate low-resolution panoramas and then upscaling to high-resolution, such as (Akimoto, Matsuo, and Aoki 2022a;Chen, Wang, and Liu 2022). These methods, however, can still result in artifacts caused by error accumulation.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

HORIZON: High-Resolution Semantically Controlled Panorama Synthesis

Yan,

Ji,

et al. 2024

AAAI

View full text Add to dashboard Cite

Panorama synthesis endeavors to craft captivating 360-degree visual landscapes, immersing users in the heart of virtual worlds. Nevertheless, contemporary panoramic synthesis techniques grapple with the challenge of semantically guiding the content generation process. Although recent breakthroughs in visual synthesis have unlocked the potential for semantic control in 2D flat images, a direct application of these methods to panorama synthesis yields distorted content. In this study, we unveil an innovative framework for generating high-resolution panoramas, adeptly addressing the issues of spherical distortion and edge discontinuity through sophisticated spherical modeling. Our pioneering approach empowers users with semantic control, harnessing both image and text inputs, while concurrently streamlining the generation of high-resolution panoramas using parallel decoding. We rigorously evaluate our methodology on a diverse array of indoor and outdoor datasets, establishing its superiority over recent related work, in terms of both quantitative and qualitative performance metrics. Our research elevates the controllability, efficiency, and fidelity of panorama synthesis to new levels.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

HORIZON: High-Resolution Semantically Controlled Panorama Synthesis

Yan,

Ji,

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Subsequently, GANs were introduced to the task [16,23,39,49] which do not suffer from the same computational limitations as pixel-wise autoregressive approaches. More recent works have utilized a Vector Quantised-Variational Autoencoder (VQ-VAE) [47] to great success [2,4]. Similar to image outpainting, our task requires generated images to appear coherent in weather and location; however, we also seek to generate distinct, partially overlapping camera views and require that portions of these views are conditionally generated from a BEV layout.…”

Section: Related Workmentioning

confidence: 99%

“…A key problem that arises when generating multiple images in parallel is the quadratic complexity of the selfattention mechanism. One solution to this issue would be to limit the sequence length of our transformer by using performing image extrapolation as in [2]. However, this limits the scene context and can cause later images to appear far different from the first image, despite having local image consistency.…”

Section: Random Maskingmentioning

confidence: 99%

Street-View Image Generation from a Bird's-Eye View Layout

Swerdlow¹,

Xu²,

Zhou³

2023

Preprint

View full text Add to dashboard Cite

Bird's-Eye View (BEV) Perception has received increasing attention in recent years as it provides a concise and unified spatial representation across views and benefits a diverse set of downstream driving applications. While the focus has been placed on discriminative tasks such as BEV segmentation, the dual generative task of creating streetview images from a BEV layout has rarely been explored. The ability to generate realistic street-view images that align with a given HD map and traffic layout is critical for visualizing complex traffic scenarios and developing robust perception models for autonomous driving. In this paper, we propose BEVGen, a conditional generative model that synthesizes a set of realistic and spatially consistent surrounding images that match the BEV layout of a traffic scenario. BEVGen incorporates a novel cross-view transformation and spatial attention design which learn the relationship between cameras and map views to ensure their consistency. Our model can accurately render road and lane lines, as well as generate traffic scenes under different weather conditions and times of day. The code will be made publicly available.

show abstract

“…Spherical panoramic images, also known as 360 • panoramic images or omnidirectional panoramic images, are used in various domains such as autonomous driving (de La Garanderie, Abarghouei, and Breckon 2018; Ma et al 2021), virtual reality (Xu, Zhang, and Gao 2021;Ai et al 2022), etc. Numerous studies (Yan et al 2022;Hara, Mukuta, and Harada 2021;Akimoto, Matsuo, and Aoki 2022) have been proposed for the synthesis of spherical panoramic images, with a primary focus on reconstructing scenes from narrow field of view (NFOV) images. However, these generation methods often produce images of inferior quality and lack controllability, which are crucial in real applications.…”

Section: Introductionmentioning

confidence: 99%

SphereDiffusion: Spherical Geometry-Aware Distortion Resilient Diffusion Model

Wu,

Li,

et al. 2024

AAAI

View full text Add to dashboard Cite

Controllable spherical panoramic image generation holds substantial applicative potential across a variety of domains. However, it remains a challenging task due to the inherent spherical distortion and geometry characteristics, resulting in low-quality content generation. In this paper, we introduce a novel framework of SphereDiffusion to address these unique challenges, for better generating high-quality and precisely controllable spherical panoramic images. For the spherical distortion characteristic, we embed the semantics of the distorted object with text encoding, then explicitly construct the relationship with text-object correspondence to better use the pre-trained knowledge of the planar images. Meanwhile, we employ a deformable technique to mitigate the semantic deviation in latent space caused by spherical distortion. For the spherical geometry characteristic, in virtue of spherical rotation invariance, we improve the data diversity and optimization objectives in the training process, enabling the model to better learn the spherical geometry characteristic. Furthermore, we enhance the denoising process of the diffusion model, enabling it to effectively use the learned geometric characteristic to ensure the boundary continuity of the generated images. With these specific techniques, experiments on Structured3D dataset show that SphereDiffusion significantly improves the quality of controllable spherical image generation and relatively reduces around 35% FID on average.

show abstract

Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creation

Cited by 18 publications

References 28 publications

HORIZON: High-Resolution Semantically Controlled Panorama Synthesis

HORIZON: High-Resolution Semantically Controlled Panorama Synthesis

Street-View Image Generation from a Bird's-Eye View Layout

SphereDiffusion: Spherical Geometry-Aware Distortion Resilient Diffusion Model

Contact Info

Product

Resources

About