Dual Learning for Cross-domain Image Captioning

Zhao, Wei; Xu, Wei; Yang, Min; Ye, Jianbo; Zhao, Zhou; Feng, Yabing; Qiao, Yu

doi:10.1145/3132847.3132920

Cited by 48 publications

(27 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One captioner adapts the sentence style from source to target domain, whereas two critics, namely domain critic and multi-modal critic, aim at distinguishing them. Zhao et al [141] fine-tuned the pre-trained source model on limited data in the target domain via a dual learning mechanism.…”

Section: G Image Captioningmentioning

confidence: 99%

Deep visual domain adaptation: A survey

2018

View full text Add to dashboard Cite

Deep domain adaptation has emerged as a new learning technique to address the lack of massive amounts of labeled data. Compared to conventional methods, which learn shared feature subspaces or reuse important source instances with shallow representations, deep domain adaptation methods leverage deep networks to learn more transferable representations by embedding domain adaptation in the pipeline of deep learning. There have been comprehensive surveys for shallow domain adaptation, but few timely reviews the emerging deep learning based methods. In this paper, we provide a comprehensive survey of deep domain adaptation methods for computer vision applications with four major contributions. First, we present a taxonomy of different deep domain adaptation scenarios according to the properties of data that define how two domains are diverged. Second, we summarize deep domain adaptation approaches into several categories based on training loss, and analyze and compare briefly the state-of-the-art methods under these categories. Third, we overview the computer vision applications that go beyond image classification, such as face recognition, semantic segmentation and object detection. Fourth, some potential deficiencies of current methods and several future directions are highlighted.

show abstract

Section: G Image Captioningmentioning

confidence: 99%

Deep visual domain adaptation: A survey

2018

View full text Add to dashboard Cite

show abstract

“…It can learn the distribution of the real dataset and generate synthetic samples conforming to that distribution. GAN have been successfully applied in image generation, image inpainting [48], image captioning [49][50][51], object detection [52], semantic segmentation [53,54], natural language processing [55,56], speech enhancement [57], credit card fraud detection [58] and supervised learning with insufficient training data [59]. From the experiments and results of these studies, it is evident that GAN conforms to the distribution of the original data samples and can generate realistic synthetic data.…”

Section: Related Workmentioning

confidence: 99%

GAN-based synthetic brain PET image generation

2020

View full text Add to dashboard Cite

In recent days, deep learning technologies have achieved tremendous success in computer vision-related tasks with the help of large-scale annotated dataset. Obtaining such dataset for medical image analysis is very challenging. Working with the limited dataset and small amount of annotated samples makes it difficult to develop a robust automated disease diagnosis model. We propose a novel approach to generate synthetic medical images using generative adversarial networks (GANs). Our proposed model can create brain PET images for three different stages of Alzheimer's disease-normal control (NC), mild cognitive impairment (MCI), and Alzheimer's disease (AD).

show abstract

“…Thus, methods developed on such datasets might not be easily adopted in the wild. Nevertheless, great efforts have been made to extend captioning to out-of-domain data [3,9,69] or different styles beyond mere factual descriptions [22,55]. In this work we explore unsupervised captioning, where image and language sources are independent.…”

Section: Language Domainmentioning

confidence: 99%

“…Chen et al [9] address cross-domain captioning, where the source domain consists of image-caption pairs and the goal is to leverage unpaired data from a target domain through a critic. In [69], the cross-domain problem is addressed with a cycle objective. Similarly, unpaired data can be used to generate stylized descriptions [22,46].…”

Section: Related Workmentioning

confidence: 99%

Towards Unsupervised Image Captioning With Shared Multimodal Embeddings

Laina

Rupprecht

Navab

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Understanding images without explicit supervision has become an important problem in computer vision. In this paper, we address image captioning by generating language descriptions of scenes without learning from annotated pairs of images and their captions. The core component of our approach is a shared latent space that is structured by visual concepts. In this space, the two modalities should be indistinguishable. A language model is first trained to encode sentences into semantically structured embeddings. Image features that are translated into this embedding space can be decoded into descriptions through the same language model, similarly to sentence embeddings. This translation is learned from weakly paired images and text using a loss robust to noisy assignments and a conditional adversarial component. Our approach allows to exploit large text corpora outside the annotated distributions of image/caption data. Our experiments show that the proposed domain alignment learns a semantically meaningful representation which outperforms previous work.

show abstract

Dual Learning for Cross-domain Image Captioning

Cited by 48 publications

References 17 publications

Deep visual domain adaptation: A survey

Deep visual domain adaptation: A survey

GAN-based synthetic brain PET image generation

Towards Unsupervised Image Captioning With Shared Multimodal Embeddings

Contact Info

Product

Resources

About