MM-AVS: A Full-Scale Dataset for Multi-modal Summarization

Abstract

Multimodal summarization becomes increasingly significant as it is the basis for question answering, Web search, and many other downstream tasks. However, its learning materials have been lacking a holistic organization by integrating resources from various modalities, thereby lagging behind the research progress of this field. In this study, we present a full-scale multimodal dataset comprehensively gathering documents, summaries, images, captions, videos, audios, transcripts, and titles in English from CNN and Daily Mail. To our best knowledge, this is the first collection that spans all modalities and nearly comprises all types of materials available in this community. In addition, we devise a baseline model based on the novel dataset, which employs a newly proposed Jump-Attention mechanism based on transcripts. The experimental results validate the important assistance role of the external information for multimodal summarization.

MHMS: Multimodal Hierarchical Multimedia Summarization

“…For VMSMO dataset, the quality of chosen cover frame is evaluated by mean average precision (MAP) and recall at position (𝑅 𝑛 @𝑘) [81,108], where (𝑅 𝑛 @𝑘) measures if the positive sample is ranked in the top 𝑘 positions of 𝑛 candidates. For Daily Mail dataset and CNN dataset, we calculate the cosine image similarity (Cos) between image references and the extracted frames from videos [22,23]. We compare our MHMS model with existing multimodal summarization, video summarization, and textual summarization approaches.…”

“…We evaluated our models on three datasets: VMSMO dataset [57], Daily Mail dataset, and CNN dataset from [22,23,57]. The popular COIN and Howto100M can not be used in our task, since they lack narrations and key-step annotation [54,80].…”

“…For the data splitting, we take the same experimental setup as [57] for the VMSMO dataset. For Daily Mail dataset and CNN dataset, we split the data by 70%, 10%, 20% for train, validation and test sets, respectively, same as [22,23].…”

“…HNNattTI [9]: HNNattTI aligned the sentences and accompanying images by using attention mechanism. M 2 SM [22,23]: M 2 SM is a multimodal summarization model with a bi-stream summarization strategy for training by sharing the ability to refine significant information from long materials in text and video summarization.…”

“…Video Temporal Segmentation. We used the same model setting as [7,65] and same data splitting setting as [22,23,57] to train the video temporal segmentation module.…”