Multimodal Summarization of User-Generated Videos

Abstract

The exponential growth of user-generated content has increased the need for efficient video summarization schemes. However, most approaches underestimate the power of aural features, while they are designed to work mainly on commercial/professional videos. In this work, we present an approach that uses both aural and visual features in order to create video summaries from user-generated videos. Our approach produces dynamic video summaries, that is, comprising the most “important” parts of the original video, which are arranged so as to preserve their temporal order. We use supervised knowledge from both the aforementioned modalities and train a binary classifier, which learns to recognize the important parts of videos. Moreover, we present a novel user-generated dataset which contains videos from several categories. Every 1 sec part of each video from our dataset has been annotated by more than three annotators as being important or not. We evaluate our approach using several classification strategies based on audio, video and fused features. Our experimental results illustrate the potential of our approach.

“In the Wild” Video Content as a Special Case of User Generated Content and a System for Its Recognition

“…The results were compared with [ 26 ]. The authors presented a different approach and identified the samples based on video and audio recognition, contrary to our method, which contains only video.…”

MCR: Multilayer cross‐fusion with reconstructor for multimodal abstractive summarisation

“…Miao et al [13] proposed an extractive model to integrate multimodal new data and extract keywords as the short title, with a reinforcement training method. Some works utilised a collection of visual, audio, and text modalities sourced from web blogs and multimedia streams to generate summaries [14][15][16][17]. Some works have been dedicated to generating multimodal output instead of only text or…”

Multimodal Abstractive Summarization using bidirectional encoder representations from transformers with attention mechanism

“…The quantity of user-generated instructional videos posted online has exponentially increased in recent years due to the growing popularity of video-sharing platforms. Due to the amount of videos available online, there is a growing need for the effective methods to find and retrieve relevant videos [ 3 , 4 ]. In order to meet these needs, numerous researchers have employed the concept of video summarization to generate brief summaries from the videos [ [5] , [6] ].…”