Multi-modal Summarization for Asynchronous Collection of Text, Image, Audio and Video

Abstract

The rapid increase in multimedia data transmission over the Internet necessitates the multi-modal summarization (MMS) from collections of text, image, audio and video. In this work, we propose an extractive multi-modal summarization method that can automatically generate a textual summary given a set of documents, images, audios and videos related to a specific topic. The key idea is to bridge the semantic gaps between multi-modal content. For audio information, we design an approach to selectively use its transcription. For visual information, we learn the joint representations of text and images using a neural network. Finally, all of the multimodal aspects are considered to generate the textual summary by maximizing the salience, non-redundancy, readability and coverage through the budgeted optimization of submodular functions. We further introduce an MMS corpus in English and Chinese, which is released to the public 1 . The experimental results obtained on this dataset demonstrate that our method outperforms other competitive baseline methods.

Exploring Methods for Predicting Important Utterances Contributing to Meeting Summarization

“…Erol et al [54] proposed a method for detecting important segments of a recorded meeting based on activity analysis, which simply measured audio amplitude and luminance difference between two video frames as well as text analysis using tf-idf. More recently, Li et al [55] proposed an extractive multi-modal summarization method that selects salient sentences by considering the images, audio, and videos related to a specific topic. However, they did not address the issue of meeting summarization.…”

Multimodal Summarization with Guidance of Multimodal Reference

“…Multimodal summarization has been proposed to extract the most important information from the multimedia information. The most significant difference between multimodal summarization (Mademlis et al 2016;Li et al 2017;2018b;Zhu et al 2018) and text summarization (Zhu et al 2017;Paulus, Xiong, and Socher 2018;Celikyilmaz et al 2018;Li et al 2018c;Zhu et al 2019) lies in whether the input data contains two or more modalities of data. One of the most significant advantages of the task is that it can use the rich information in multimedia data to improve the quality of the final summary.…”

Text-Image-Video Summary Generation Using Joint Integer Linear Programming

“…A few deep learning frameworks [2,11,31] show promising results, too. Li et al [12] uses an asynchronous dataset containing text, images and videos to generate a textual summary. Although some work on document summarization has been done using ILP, to the best of our knowledge no one has ever used an ILP framework in the area of multi-modal summarization.…”

“…There is no benchmark dataset for the TIVS task. Therefore, we created our own text-image-video dataset by extending and manually annotating the multi-modal summarization dataset introduced by Li et al [12]. Their dataset comprised of 25 new topics.…”

“…To sum up, we make the following contributions: (1) We present a novel multimodal summarization task which takes news with images and videos as input, and outputs text, images and video as summary. (2) We create an extension of the multi-modal summarization dataset [12] by constructing multi-modal references containing text, images and video for each topic. (3) We design a joint ILP framework to address the proposed multi-modal summarization task.…”