Image to Modern Chinese Poetry Creation via a Constrained Topic-aware Model

Abstract

Artificial creativity has attracted increasing research attention in the field of multimedia and artificial intelligence. Despite the promising work on poetry/painting/music generation, creating modern Chinese poetry from images, which can significantly enrich the functionality of photo-sharing platforms, has rarely been explored. Moreover, existing generation models cannot tackle three challenges in this task: (1) Maintaining semantic consistency between images and poems; (2) preventing topic drift in the generation; (3) avoidance of certain words appearing frequently. These three points are even common challenges in other sequence generation tasks. In this article, we propose a Constrained Topic-aware Model (CTAM) to create modern Chinese poetries from images regarding the challenges above. Without image-poetry paired dataset, we construct a visual semantic vector to embed visual contents via image captions. For the topic-drift problem, we propose a topic-aware poetry generation model. Additionally, we design an Anti-frequency Decoding (AFD) scheme to constrain high-frequency characters in the generation. Experimental results show that our model achieves promising performance and is effective in poetry’s readability and semantic consistency.

Images2Poem in different contexts with Dual‐CharRNN

“…Poetry is supposed to be oriented to the public, and good poetry should be accepted and liked by the public. Referring to [23, 30, 42], we use three evaluation standards to evaluate the quality of generated pomes, which are semantic consistency, readability, and aesthetics: Semantic consistency (S): Whether the theme expressed in the poetry corresponds to the image. Readability (R): Whether a single sentence of the poetry is fluent, and whether all sentences are coherent. Aesthetics (A): Whether the poetry expresses aesthetics and emotions poetically. …”

“…Constrained Topic‐aware Model (CTAM) [30]: It constructs a visual semantic vector to embed visual content through the image CTAM to create modern Chinese poetry.…”

“…To create poetry related to the given image, we need to mine semantic information from the visual image [30], because an image not only contains the visible objects, but also the relationships between them. These objects play a crucial role in the content of the poetry, and their relationships influence the creator's emotions or the context of the poetry.…”

When Creators Meet the Metaverse: A Survey on Computational Arts

“…We have to understand the reasons for topic drift throughout the automatic generation process, and to design preventive mechanisms, perhaps more transparent and explainable tools are required. Also, some words appear more frequently than the rest, but they could deteriorate the model effectiveness [213]. On the other hand, we would like to emphasise that most automatically generated poetry are considered as rigid outputs.…”

iPoet: interactive painting poetry creation with visual multimodal analysis

“…Since there is no high-quality large-scale dataset of image-poem pairs, most works are mainly restricted to two different strategies to train the poem generation models. The first uses the keywords extracted from the image to generate poems (Cheng et al 2018;Wu et al 2020). The extraction and generation models are trained separately on their own datasets.…”