Hierarchical video content description and summarization using unified semantic and visual similarity

Abstract

Video is increasingly the medium of choice for a variety of communication channels, resulting primarily from increased levels of networked multimedia systems. One way to keep our heads above the video sea is to provide summaries in a more tractable format. Many existing approaches are limited to exploring important low-level feature related units for summarization. Unfortunately, the semantics, content and structure of the video do not correspond to low-level features directly, even with closed-captions, scene detection, and audio signal processing. The drawbacks of existing methods are the following:(1) instead of unfolding semantics and structures within the video, low-level units usually address only the details, and (2) any important unit selection strategy based on low-level features cannot be applied to general videos. Providing users with an overview of the video content at various levels of summarization is essential for more efficient database retrieval and browsing. In this paper, we present a hierarchical video content description and summarization strategy supported by a novel joint semantic and visual similarity strategy. To describe the video content efficiently and accurately, a video content description ontology is adopted. Various video processing techniques are then utilized to construct a semi-automatic video annotation framework. By integrating acquired content description data, a hierarchical video content structure is constructed with group merging and clustering. Finally, a four layer video summary with different granularities is assembled to assist users in unfolding the video content in a progressive way. Experiments on real-word videos have validated the effectiveness of the proposed approach.

Novelty detection for cross-lingual news stories with visual duplicates and speech transcripts

“…Hsu et al [9] tracked four topics with visual duplicates and semantic concepts, and found that near-duplicates significantly improved the tracking performance. Zhu et al [35] presented a hierarchical video content description and summarization strategy supported by a joint textual and visual similarity. Zhai et al [31] linked news stories by combining keyframe matching and textual correlation.…”

Hierarchical structuring of video previews by Leading-Cluster-Analysis

“…An interesting example of hierarchical agglomerative clustering using a variant of the average-linkage criterion is the semi-automatic approach proposed in [21], whose final aim is to extract a hierarchical summary. In this work similarity between shots is defined on arbitrarily chosen key-frames in terms of HSV color histogram and texture features.…”

“…The two measures proposed in [21] are used for evaluating each layer of the preview. The first one measures the clustering precision P (answering the question "how accurate is this summary layer?")…”

Multimodal News Story Clustering With Pairwise Visual Near-Duplicate Constraint

“…Duygulu et al [12] presented the technique to mine and track the repeated sequence of shots. A hierarchical video content description and summarization strategy was proposed in [40] with the support of a joint textual and visual similarity. Zhai et al [37] linked news stories by combining keyframe matching and textual correlation.…”