Unsupervised Image Captioning

Abstract

Deep neural networks have achieved great successes on the image captioning task. However, most of the existing models depend heavily on paired image-sentence datasets, which are very expensive to acquire. In this paper, we make the first attempt to train an image captioning model in an unsupervised manner. Instead of relying on manually labeled image-sentence pairs, our proposed model merely requires an image set, a sentence corpus, and an existing visual concept detector. The sentence corpus is used to teach the captioning model how to generate plausible sentences. Meanwhile, the knowledge in the visual concept detector is distilled into the captioning model to guide the model to recognize the visual concepts in an image. In order to further encourage the generated captions to be semantically consistent with the image, the image and caption are projected into a common latent space so that they can reconstruct each other. Given that the existing sentence corpora are mainly designed for linguistic research and are thus with little reference to image contents, we crawl a large-scale image description corpus of two million natural sentences to facilitate the unsupervised image captioning scenario. Experimental results show that our proposed model is able to produce quite promising results without any caption annotations.Recently, there have been several attempts at relaxing the reliance on paired image-sentence data for image captioning training. As shown in Figure 1 (b), Hendricks et al. [3] proposed to generate captions for novel objects, which are not present in the paired image-caption training data but exist in image recognition datasets, e.g., ImageNet. As such, novel object information can be introduced into the gener-

Towards Unsupervised Image Captioning With Shared Multimodal Embeddings

“…When language and images come from different sources, some weak supervisory signal is needed to align the manifold of visual concepts to the textual domain. Similar to previous work [18], we use a pre-trained object detector to generate an initial noisy alignment between the text source and visual entities that can be detected in the image.…”

“…Most closely related to our work is [18] which does not require any image-sentence pairs. In this case, it is optimal to use a language domain which is rich in visual concepts.…”

Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach

“…We indicate the ablation study by: (A) the usage of the proposed GAN that distinguishes real or fake image-caption pairs, (B) pseudo-labeling, and (C) noise handling by sample re-weighting. We also compare with [Gu et al, 2018] and [Feng et al, 2019], which are trained with unpaired datasets.…”

“…We also compare the recent unpaired image captioning methods [Gu et al, 2018;Feng et al, 2019] in Table 1. Both of the methods are evaluated on MS COCO testset.…”

Generating Classical Chinese Poems from Vernacular Chinese

“…The unsupervised machine translation framework is also applied to various other tasks, e.g. image captioning (Feng et al, 2019), text style transfer (Zhang et al, 2018), speech to text translation (Bansal et al, 2017) and clinical text simplification (Weng et al, 2019). The UMT framework makes it possible to apply neural models to tasks where limited human labeled data is available.…”