Topic Adaptation and Prototype Encoding for Few-Shot Visual Storytelling

Abstract

Visual Storytelling (VIST) is a task to tell a narrative story about a certain topic according to the given photo stream. The existing studies focus on designing complex models, which rely on a huge amount of human-annotated data. However, the annotation of VIST is extremely costly and many topics cannot be covered in the training dataset due to the long-tail topic distribution. In this paper, we focus on enhancing the generalization ability of the VIST model by considering the few-shot setting. Inspired by the way humans tell a story, we propose a topic adaptive storyteller to model the ability of inter-topic generalization. In practice, we apply the gradientbased meta-learning algorithm on multi-modal seq2seq models to endow the model the ability to adapt quickly from topic to topic. Besides, We further propose a prototype encoding structure to model the ability of intra-topic derivation. Specifically, we encode and restore the few training story text to serve as a reference to guide the generation at inference time. Experimental results show that topic adaptation and prototype encoding structure mutually bring benefit to the few-shot model on BLEU and METEOR metric. The further case study shows that the stories generated after few-shot adaptation are more relative and expressive. CCS CONCEPTS • Computing methodologies → Natural language generation; Computer vision tasks.

A Survey on Long-Tailed Visual Recognition

“…6.1, data naturally satisfy the long-tailed distribution in many fields, so solving the long-tailed problem may be able to improve the performance of models in these fields. However, according to the available research results, only a few work have studied from the perspective of the long-tailed distribution in their research field, such as long-tailed distribution of object classes in UAV images [179], long-tailed distribution of driving behavior in autonomous driving [116] , and topic in the field of visual story telling [91], content-related words for video captioning tasks [192], pose inclusion in datasets for 3D human pose estimation [181], dermatological categories in dermatological diagnosis [123], etc. We believe that for many research fields, the existing work to analyze and solve the long-tailed distribution problem is still not enough.…”

Latent Memory-augmented Graph Transformer for Visual Storytelling

“…For example, Kim et al [27] introduced two levels of hierarchical RNNs with attention mechanisms in terms of global encoding level and local image level to address multiimage cued story generation. On the other hand, several recent works [1,18,19,33,34,37,62,67,70] have devoted to incorporating semantic knowledge to improve the quality of the generated story. For instance, Li et al [33] inferred semantic concepts and captured cross-modal rules for visual storytelling, and Hsu et al [19] distilled a wealthy of words from an external knowledge graph to generate more interesting stories.…”

“…With the popularity of social networks, a tremendous number of users routinely share a series of photos, along with their related comments/stories on social media platforms such as Instagram and Flickr. Consequently, a new task of visual storytelling [1,24,27,34,37,43,62,64,72], which aims at automatically generating a narrative story for an image stream (as shown in Figure 1), has recently attracted increasing attention in the multimedia community. Given a stream of images, humans are capable of composing a suitable story line and then generating a sequence of sentences.…”