Video storytelling based on gated video memorability filtering

Abstract

Video storytelling aims to generate a successive and succinct paragraph composed of multiple sentences to describe the visual contents in a long video. The major difficulty with video storytelling is the conflict between complex visual semantics and concise textual description. Existing methods address this problem by applying different segmentation strategies to long videos. However, these methods ignore that some story-irrelevant visual contents could be considered as noises, so visual information in a long video is not always conducive to story generation. Therefore, we propose a novel video storytelling model based on gated video memorability filtering (GVMF), which generates highquality textual descriptions by filtering out clips with lower memorability scores, where objects and characters appear less frequently. The reason for introducing video memorability prediction is that video clips with higher video memorability scores tend to have more characters and objects so tend to be noticed by annotators for writing sentences. Experimental results demonstrate the effectiveness and competitiveness of our method, which achieves the new state-of-the-art results on Video Story dataset.