Video co-summarization: Video summarization by visual co-occurrence

Abstract

We present video co-summarization, a novel perspective to video summarization that exploits visual co-occurrence across multiple videos. Motivated by the observation that important visual concepts tend to appear repeatedly across videos of the same topic, we propose to summarize a video by finding shots that co-occur most frequently across videos collected using a topic keyword. The main technical challenge is dealing with the sparsity of co-occurring patterns, out of hundreds to possibly thousands of irrelevant shots in videos being considered. To deal with this challenge, we developed a Maximal Biclique Finding (MBF) algorithm that is optimized to find sparsely co-occurring patterns, discarding less co-occurring patterns even if they are dominant in one video. Our algorithm is parallelizable with closed-form updates, thus can easily scale up to handle a large number of videos simultaneously. We demonstrate the effectiveness of our approach on motion capture and self-compiled YouTube datasets. Our results suggest that summaries generated by visual co-occurrence tend to match more closely with human generated summaries, when compared to several popular unsupervised techniques.

Hierarchical Recurrent Neural Network for Video Summarization

“…As in [5], the frames are initially clustered in sequential order, as consecutive frames are similar and more probably to be allocated to the same cluster. Other works construct more comprehensive models based on the idea of clustering [3,22]. As in [22], the video is transformed into an undirected graph, and the summary is generated by partitioning this graph into clusters.…”

A General Framework for Edited Video and Raw Video Summarization

“…Besides, clustering is combined with other techniques to generate a summary, such as scene recognition in [14]. Recently, a co-clusters approach is proposed in [28], which summarizes several videos of the same topic simultaneously by identifying similar shots shared across these videos.…”

Unsupervised Synchrony Discovery in Human Interaction

“…To describe the static and dynamic information of a video segment, we extract two types of features as the segment-level feature mapping [8, 19]: observation features extracted from a single frame, and interaction features extracted from two consecutive frames. Suppose the j th frame is described as a feature vector x j .…”

“…As a result, we represent a video segment X i = { x bi , ..., x ei } between the $b_i^{th}$ and the $e_i^{th}$ frames by normalizing the sum of the concatenation of the two features, resulting in a feature vector ${\varphi }_{{\mathrm{boldX}}_i}={\sum }_{j=b_i}^{e_i}[{\varphi }^{\mathrm{obs}}\left({\mathrm{boldx}}_j\right);{\varphi }^{\mathrm{int}}\left({\mathrm{boldx}}_j\right)]$. See [8,19] for details about the feature mapping.…”