A General Framework for Edited Video and Raw Video Summarization

Abstract

In this paper, we build a general summarization framework for both of edited video and raw video summarization. Overall, our work can be divided into three folds. 1) Four models are designed to capture the properties of video summaries, i.e., containing important people and objects (importance), representative to the video content (representativeness), no similar key-shots (diversity), and smoothness of the storyline (storyness). Specifically, these models are applicable to both edited videos and raw videos. 2) A comprehensive score function is built with the weighted combination of the aforementioned four models. Note that the weights of the four models in the score function, denoted as property-weight, are learned in a supervised manner. Besides, the property-weights are learned for edited videos and raw videos, respectively. 3) The training set is constructed with both edited videos and raw videos in order to make up the lack of training data. Particularly, each training video is equipped with a pair of mixing-coefficients, which can reduce the structure mess in the training set caused by the rough mixture. We test our framework on three data sets, including edited videos, short raw videos, and long raw videos. Experimental results have verified the effectiveness of the proposed framework.

Video Summarization: How to Use Deep-Learned Features Without a Large-Scale Dataset

“…As a whole, eleven baselines are employed here for comparison, including: STIMO [26], VSUMM [10], SumTransfer [7], SUM-GAN [13], SeqDPP [6], LSTM [15], TVSum [8], Li et al [14], MSDS-CC [27], LLR-SDS [28], and Online Motion AE [9]. The results from all of the baselines [26,10,6,7,13,14,15,8,27,28,9] are obtained from those reported in their papers.…”

“…Lastly, we compare the proposed method with seven baselines on the TVSum dataset, including, SUM-GAN [13], LSTM [15], TVSum [8], Li. et al [14], MSDS-CC [27], LLR-SDS [28], and Online Motion AE [9] as shown in Table III. Our approach which combines the temporal encoding by rank pooling, near-duplicate frame removal by ITQ and peaksearching by SNIP can attain the best performance.…”

“…Note that in the above simulations, many works like SUM-GAN [13], Online Motion AE [9], LLR-SDS [28], MSDS-CC [27], Li et al [14] and LSTM [15] are all based on the deep-learned features. They, however, require a large-scale annotated training videos to attain their best performance.…”

“…Recently, with the success of Convolutional Neural Networks (CNN), Mahaseni et al [13] used the generative adversarial network (GAN) which consisted of a summarizer and a discriminator to perform the unsupervised video summarization. Li et al [14] proposed a model to summarize the videos based on a weighted combination of the importance, representativeness, diversity and smoothness of the storylinemeasures. Zhang et al [15] estimated the range of inter-dependencies among frames using long-short term memory (LSTM) to conduct supervised learning for video summarization.…”

Fusing Multimodal Video Data for Detecting Moving Objects/Targets in Challenging Indoor and Outdoor Scenes

“…Certain processing pipelines have also been proposed to address the changing environmental illumination conditions [2]. Scene recognition and video summarization have been also proposed based on machine learning techniques and RGB data [6,7].…”