Semantic Storytelling Automation: A Context-Aware and Metadata-Driven Approach

Viana, Paula; Carvalho, Pedro; Andrade, Maria Teresa; Jonker, Pieter; Papanikolaou, Vasileios; Teixeira, Inês; Vilaça, Luís; Pinto, José P.; Costa, Tiago

doi:10.1145/3394171.3416528

Cited by 3 publications

(2 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, Kim et al [27] introduced two levels of hierarchical RNNs with attention mechanisms in terms of global encoding level and local image level to address multiimage cued story generation. On the other hand, several recent works [1,18,19,33,34,37,62,67,70] have devoted to incorporating semantic knowledge to improve the quality of the generated story. For instance, Li et al [33] inferred semantic concepts and captured cross-modal rules for visual storytelling, and Hsu et al [19] distilled a wealthy of words from an external knowledge graph to generate more interesting stories.…”

Section: Related Workmentioning

confidence: 99%

“…With the popularity of social networks, a tremendous number of users routinely share a series of photos, along with their related comments/stories on social media platforms such as Instagram and Flickr. Consequently, a new task of visual storytelling [1,24,27,34,37,43,62,64,72], which aims at automatically generating a narrative story for an image stream (as shown in Figure 1), has recently attracted increasing attention in the multimedia community. Given a stream of images, humans are capable of composing a suitable story line and then generating a sequence of sentences.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Latent Memory-augmented Graph Transformer for Visual Storytelling

Qin

Huang

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Visual storytelling aims to automatically generate a human-like short story given an image stream. Most existing works utilize either scene-level or object-level representations, neglecting the interaction among objects in each image and the sequential dependency between consecutive images. In this paper, we present a novel Latent Memory-augmented Graph Transformer (LMGT), a Transformer based framework for visual story generation. LMGT directly inherits the merits from the Transformer, which is further enhanced with two carefully designed components, i.e., a graph encoding module and a latent memory unit. Specifically, the graph encoding module exploits the semantic relationships among image regions and attentively aggregates critical visual features based on the parsed scene graphs. Furthermore, to better preserve inter-sentence coherence and topic consistency, we introduce an augmented latent memory unit that learns and records highly summarized latent information as the story line from the image stream and the sentence history. Experimental results on three widely-used datasets demonstrate the superior performance of LMGT over the state-of-the-art methods. CCS CONCEPTS• Computing methodologies → Natural language generation; Scene understanding.

show abstract