Learning from Simulated and Unsupervised Images through Adversarial Training

Shrivastava, Ashish; Pfister, Tomas; Tuzel, Oncel; Susskind, Josh; Wang, Wenda; Webb, Russ

doi:10.48550/arxiv.1612.07828

Cited by 58 publications

(97 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Theorem 2.1 is the foundation of many recent works on unsupervised domain adaptation via learning invariant representations [Ajakan et al, 2014, Ganin et al, 2016, Zhao et al, 2018b, Pei et al, 2018, Zhao et al, 2018a. It has also inspired various applications of domain adaptation with adversarial learning, e.g., video analysis [Hoffman et al, 2016, Shrivastava et al, 2016, natural language understanding [Zhang et al, 2017, Fu et al, 2017, speech recognition [Zhao et al, 2019a, Hosseini-Asl et al, 2018, to name a few. At a high level, the key idea is to learn a rich and parametrized feature transformation g : X → Z such that the induced source and target distributions (on Z) are close, as measured by the Hdivergence.…”

Section: A Theoretical Model For Domain Adaptationmentioning

confidence: 99%

On Learning Invariant Representation for Domain Adaptation

Zhao,

Combes,

Zhang

et al. 2019

Preprint

View full text Add to dashboard Cite

Due to the ability of deep neural nets to learn rich representations, recent advances in unsupervised domain adaptation have focused on learning domain-invariant features that achieve a small error on the source domain. The hope is that the learnt representation, together with the hypothesis learnt from the source domain, can generalize to the target domain. In this paper, we first construct a simple counterexample showing that, contrary to common belief, the above conditions are not sufficient to guarantee successful domain adaptation. In particular, the counterexample exhibits conditional shift: the class-conditional distributions of input features change between source and target domains. To give a sufficient condition for domain adaptation, we propose a natural and interpretable generalization upper bound that explicitly takes into account the aforementioned shift. Moreover, we shed new light on the problem by proving an information-theoretic lower bound on the joint error of any domain adaptation method that attempts to learn invariant representations. Our result characterizes a fundamental tradeoff between learning invariant representations and achieving small joint error on both domains when the marginal label distributions differ from source to target. Finally, we conduct experiments on real-world datasets that corroborate our theoretical findings. We believe these insights are helpful in guiding the future design of domain adaptation and representation learning algorithms.

show abstract

Section: A Theoretical Model For Domain Adaptationmentioning

confidence: 99%

On Learning Invariant Representation for Domain Adaptation

Zhao,

Combes,

Zhang

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…In their seminal work, Shrivastava et al [8] developed a CGAN conditioned on artificial images of eyes with gaze di- rection semantic information that produced realistic-looking eye images while maintaining this contextual information. They introduced a novel self-regularization term that minimized the pixel-to-pixel l 1 norm in a learned feature space between the contextual input image and the resulting filtered image.…”

Section: Related Workmentioning

confidence: 99%

Self-Attending Task Generative Adversarial Network for Realistic Satellite Image Creation

Toner¹,

Fletcher²

2021

Preprint

View full text Add to dashboard Cite

We introduce a self-attending task generative adversarial network (SATGAN) and apply it to the problem of augmenting synthetic high contrast scientific imagery of resident space objects with realistic noise patterns and sensor characteristics learned from collected data. Augmenting these synthetic data is challenging due to the highly localized nature of semantic content in the data that must be preserved. Real collected images are used to train a network what a given class of sensor's images should look like. The trained network then acts as a filter on noiseless context images and outputs realisticlooking fakes with semantic content unaltered. The architecture is inspired by conditional GANs but is modified to include a task network that preserves semantic information through augmentation. Additionally, the architecture is shown to reduce instances of hallucinatory objects or obfuscation of semantic content in context images representing space observation scenes.

show abstract

“…Recently, adversarial domain adaptation based on GANs (Goodfellow et al, 2014) have shown encouraging results for unsupervised domain adaptation directly at the pixel level. These techniques learn a generative model for source-to-target image translation, including from and to multiple domains (Taigman et al, 2016;Shrivastava et al, 2016;Zhu et al, 2017;Isola et al, 2017;Kim et al, 2017). In particular, CycleGAN (Zhu et al, 2017) leverages cycle consistency using a forward GAN and a backward GAN to improve the training stability and performance of image-to-image translation.…”

Section: Related Workmentioning

confidence: 99%

“…Several related works propose GAN-based unsupervised domain adaptation methods to address the specific domain gap between synthetic and real-world images. SimGAN (Shrivastava et al, 2016) leverages simulation for the automatic generation of large annotated datasets with the goal of refining synthetic images to make them look more realistic. Sadat Saleh et al (2018) effectively leverages synthetic data by treating foreground and background in different manners.…”

Section: Related Workmentioning

confidence: 99%

“…This is often addressed by manually labeling some amount of real-world target data to train the model on mixed synthetic and real-world labeled data (supervised domain adaptation). In contrast, several recent unsupervised domain adaptation algorithms have leveraged the potential of Generative Adversarial Networks (GANs) (Goodfellow et al, 2014) for pixel-level adaptation in this context (Bousmalis et al, 2017;Shrivastava et al, 2016). These methods often use simulators as black-box generators of (x, y) input / output training samples for the desired task.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SPIGAN: Privileged Adversarial Learning from Simulation

Lee¹,

Ros²,

Li³

et al. 2018

Preprint

View full text Add to dashboard Cite

Deep Learning for Computer Vision depends mainly on the source of supervision. Photo-realistic simulators can generate large-scale automatically labeled synthetic data, but introduce a domain gap negatively impacting performance. We propose a new unsupervised domain adaptation algorithm, called SPIGAN, relying on Simulator Privileged Information (PI) and Generative Adversarial Networks (GAN). We use internal data from the simulator as PI during the training of a target task network. We experimentally evaluate our approach on semantic segmentation. We train the networks on real-world Cityscapes and Vistas datasets, using only unlabeled real-world images and synthetic labeled data with z-buffer (depth) PI from the SYNTHIA dataset. Our method improves over no adaptation and state-of-theart unsupervised domain adaptation techniques. Figure 1: SPIGAN example inputs and outputs. From left to right: input images from a simulator; adapted images from SPIGAN's generator network; predictions from SPIGAN's privileged network (depth layers); semantic segmentation predictions from the target task network.

show abstract

Learning from Simulated and Unsupervised Images through Adversarial Training

Cited by 58 publications

References 0 publications

On Learning Invariant Representation for Domain Adaptation

On Learning Invariant Representation for Domain Adaptation

Self-Attending Task Generative Adversarial Network for Realistic Satellite Image Creation

SPIGAN: Privileged Adversarial Learning from Simulation

Contact Info

Product

Resources

About