Beyond Narrative Description

Abstract

Automatic generation of natural language from images has attracted extensive attention. In this paper, we take one step further to investigate generation of poetic language (with multiple lines) to an image for automatic poetry creation. This task involves multiple challenges, including discovering poetic clues from the image (e.g., hope from green), and generating poems to satisfy both relevance to the image and poeticness in language level. To solve the above challenges, we formulate the task of poem generation into two correlated sub-tasks by multi-adversarial training via policy gradient, through which the cross-modal relevance and poetic language style can be ensured. To extract poetic clues from images, we propose to learn a deep coupled visual-poetic embedding, in which the poetic representation from objects, sentiments 1 and scenes in an image can be jointly learned. Two discriminative networks are further introduced to guide the poem generation, including a multi-modal discriminator and a poem-style discriminator. To facilitate the research, we have released two poem datasets by human annotators with two distinct properties: 1) the first human annotated imageto-poem pair dataset (with 8, 292 pairs in total), and 2) to-date the largest public English poem corpus dataset (with 92, 265 different poems in total). Extensive experiments are conducted with 8K images, among which 1.5K image are randomly picked for evaluation. Both objective and subjective evaluations show the superior performances against the state-of-the-art methods for poem generation from images. Turing test carried out with over 500 human subjects, among which 30 evaluators are poetry experts, demonstrates the effectiveness of our approach. * This work was conducted when Bei Liu was a research intern at Microsoft Research. † Corresponding author 1 We consider both adjectives and verbs that can express emotions and feelings as sentiment words in this research.

AI Ekphrasis: Multi-Modal Learning with Foundation Models for Fine-Grained Poetry Retrieval

“…Liu B. et al [27] present a novel approach to the image-based poetry generation problem, by incorporating a CNN-based deep-coupled visual-poetic embedding model for the object, sentiment, and scene features. This is followed by RNN-based adversarial training with multi-discriminators as rewards for policy gradient.…”

“…The MultiM-Poem dataset is a collection of 8,292 image and poem pairs [27]. It is a subset of a larger dataset (multiM-Poem-Ex, size = 26,161 pairs) collected from the internet, targeting human-written free-form poems to illustrate paired images.…”

Prose2Poem: The Blessing of Transformers in Translating Prose to Persian Poetry

“…It is worth mentioning that a couple of studies have addressed the challenge of Poetry Generation in different settings. For instance, whereas Liu et al (2018) and(Xu et al, 2018) each intro-1 https://github.com/... duced an approach of generating poems based on visual poetic clues in English and Chinese, (Loller-Andersen and Gambäck, 2018) developed the same but intended to better satisfied rhythmical expectations.…”

Generative Adversarial Networks: A Literature Review

“…Then, GAWWN [100] was proposed by this team to improve the accuracy of the generated images. In terms of image to text, Liu et al proposed the method of generating poetry through images, which for the first-time incorporated image processing and poetry generation into a framework, making the cognition of machines have the ability to approach human beings [101].…”