ARShadowGAN: Shadow Generative Adversarial Network for Augmented Reality in Single Light Scenes

Liu, Daquan; Long, Chengjiang; Zhang, Hong-Pan; Yu, Hanning; Dong, Xinzhi; Xiao, Chunxia

doi:10.1109/cvpr42600.2020.00816

Cited by 107 publications

(77 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, generative adversarial networks are useful tools to directly create artificial content. They also can add missing shadows to marked virtual shadowless objects in real scene images (Liu et al, 2020) and output a complete AR scene image. AR content can also be created with image-based lighting (IBL) renderers, which utilise environment textures containing the illumination details of a scene.…”

Section: State Of the Artmentioning

confidence: 99%

Deep Light Direction Reconstruction from single RGB images

Miller¹,

Nischwitz²,

Westermann³

2021

CSRN

View full text Add to dashboard Cite

In augmented reality applications, consistent illumination between virtual and real objects is important for creating an immersive user experience. Consistent illumination can be achieved by appropriate parameterisation of the virtual illumination model, that is consistent with real-world lighting conditions. In this study, we developed a method to reconstruct the general light direction from red-green-blue (RGB) images of real-world scenes using a modified VGG-16 neural network. We reconstructed the general light direction as azimuth and elevation angles. To avoid inaccurate results caused by coordinate uncertainty occurring at steep elevation angles, we further introduced stereographically projected coordinates. Unlike recent deep-learning-based approaches for reconstructing the light source direction, our approach does not require depth information and thus does not rely on special red-green-blue- depth (RGB-D) images as input.

show abstract

Section: State Of the Artmentioning

confidence: 99%

Deep Light Direction Reconstruction from single RGB images

Miller¹,

Nischwitz²,

Westermann³

2021

CSRN

View full text Add to dashboard Cite

show abstract

“…Generative Adversarial Networks (GANs) were developed by reference [33] and gained popularity due to their applicability in a variety of fields. Applications include augmented reality, data generation and data augmentation [34][35][36]. A comprehensive review of research towards GANs from recent years can be found in reference [37].…”

Section: Generative Adversarial Networkmentioning

confidence: 99%

Post-Disaster Building Damage Detection from Earth Observation Imagery Using Unsupervised and Transferable Anomaly Detecting Generative Adversarial Networks

et al. 2020

View full text Add to dashboard Cite

We present an unsupervised deep learning approach for post-disaster building damage detection that can transfer to different typologies of damage or geographical locations. Previous advances in this direction were limited by insufficient qualitative training data. We propose to use a state-of-the-art Anomaly Detecting Generative Adversarial Network (ADGAN) because it only requires pre-event imagery of buildings in their undamaged state. This approach aids the post-disaster response phase because the model can be developed in the pre-event phase and rapidly deployed in the post-event phase. We used the xBD dataset, containing pre- and post- event satellite imagery of several disaster-types, and a custom made Unmanned Aerial Vehicle (UAV) dataset, containing post-earthquake imagery. Results showed that models trained on UAV-imagery were capable of detecting earthquake-induced damage. The best performing model for European locations obtained a recall, precision and F1-score of 0.59, 0.97 and 0.74, respectively. Models trained on satellite imagery were capable of detecting damage on the condition that the training dataset was void of vegetation and shadows. In this manner, the best performing model for (wild)fire events yielded a recall, precision and F1-score of 0.78, 0.99 and 0.87, respectively. Compared to other supervised and/or multi-epoch approaches, our results are encouraging. Moreover, in addition to image classifications, we show how contextual information can be used to create detailed damage maps without the need of a dedicated multi-task deep learning framework. Finally, we formulate practical guidelines to apply this single-epoch and unsupervised method to real-world applications.

show abstract

“…Image composition targets at copying a foreground object from one image and pasting it on another background image to produce a composite image. In recent years, image composition has drawn increasing attention from a wide range of applications in the fields of medical science, education, and entertainment [1,46,23]. Some deep learning methods [20,3,30,2] have been developed to improve the realism of composite image in terms of color consistency, relative scaling, spatial layout, occlusion, and viewpoint transformation.…”

Section: Introductionmentioning

confidence: 99%

Shadow Generation for Composite Image in Real-world Scenes

Yan¹,

Niu²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Image composition targets at inserting a foreground object on a background image. Most previous image composition methods focus on adjusting the foreground to make it compatible with background while ignoring the shadow effect of foreground on the background. In this work, we focus on generating plausible shadow for the foreground object in the composite image. First, we contribute a real-world shadow generation dataset DESOBA by generating synthetic composite images based on paired real images and deshadowed images. Then, we propose a novel shadow generation network SGRNet, which consists of a shadow mask prediction stage and a shadow filling stage. In the shadow mask prediction stage, foreground and background information are thoroughly interacted to generate foreground shadow mask. In the shadow filling stage, shadow parameters are predicted to fill the shadow area. Extensive experiments on our DESOBA dataset and real composite images demonstrate the effectiveness of our proposed method.

show abstract

ARShadowGAN: Shadow Generative Adversarial Network for Augmented Reality in Single Light Scenes

Cited by 107 publications

References 40 publications

Deep Light Direction Reconstruction from single RGB images

Deep Light Direction Reconstruction from single RGB images

Post-Disaster Building Damage Detection from Earth Observation Imagery Using Unsupervised and Transferable Anomaly Detecting Generative Adversarial Networks

Shadow Generation for Composite Image in Real-world Scenes

Contact Info

Product

Resources

About