SemStyle: Learning to Generate Stylised Image Captions Using Unaligned Text

Abstract

Linguistic style is an essential part of written communication, with the power to affect both clarity and attractiveness. With recent advances in vision and language, we can start to tackle the problem of generating image captions that are both visually grounded and appropriately styled. Existing approaches either require styled training captions aligned to images or generate captions with low relevance. We develop a model that learns to generate visually relevant styled captions from a large corpus of styled text without aligned images. The core idea of this model, called SemStyle, is to separate semantics and style. One key component is a novel and concise semantic term representation generated using natural language processing techniques and frame semantics. In addition, we develop a unified language model that decodes sentences with diverse word choices and syntax for different styles. Evaluations, both automatic and manual, show captions from SemStyle preserve image semantics, are descriptive, and are style shifted. More broadly, this work provides possibilities to learn richer image descriptions from the plethora of linguistic data available on the web.

Towards Unsupervised Image Captioning With Shared Multimodal Embeddings

“…In [69], the cross-domain problem is addressed with a cycle objective. Similarly, unpaired data can be used to generate stylized descriptions [22,46]. Anderson et al [3] propose a method to complete partial sequence data, e.g.…”

Dixit: Interactive Visual Storytelling via Term Manipulation

“…Alexander et al propose two-stage style transfer for image captioning. They first extract objects and verbs from an image and generate a stylish caption using an RNN trained on story corpora [6]. However, each such caption is only a single story-like sentence and is independent of other captions; put together, the captions do not constitute a context-coherent story.…”

“…To train the term prediction model, we extract terms from captions in the COCO dataset [5] as the gold labels for the first stage to identity terms in images. The selection of terms is inspired by Semstyle [6]. Each sentence is represented as a combination of several noun (objects) and verb (actions) terms, which preserves the most crucial information needed to generate stories.…”

“…described in Section 4, the LSTM model is trained with images and terms extracted from their corresponding captions in the COCO dataset; this pipeline is similar to that depicted in [6].…”

Aesthetic Image Captioning From Weakly-Labelled Photographs

“…Availability of large curated datasets such as MS-COCO [41] (100K images), Flickr30K [64] (30K images) or Conceptual Captions [74] (3M images) made it possible to train deep learning models for complex, multi-modal tasks such as natural image captioning (NIC) [81] where the goal is to factually describe the image content. Similarly, several other captioning variants such as visual question answering [5], visual storytelling [38], stylized captioning [56] have also been explored. Recently, the PCCD dataset (∼ 4200 images) [11] opened up a new area of research of describing images aesthetically.…”