Multimodal News Story Clustering With Pairwise Visual Near-Duplicate Constraint

Abstract

Abstract-Story clustering is a critical step for news retrieval, topic mining, and summarization. Nonetheless, the task remains highly challenging owing to the fact that news topics exhibit clusters of varying densities, shapes, and sizes. Traditional algorithms are found to be ineffective in mining these types of clusters. This paper offers a new perspective by exploring the pairwise visual cues deriving from near-duplicate keyframes (NDK) for constraint-based clustering. We propose a constraint-driven co-clustering algorithm (CCC), which utilizes the near-duplicate constraints built on top of text, to mine topic-related stories and the outliers. With CCC, the duality between stories and their underlying multimodal features is exploited to transform features in low-dimensional space with normalized cut. The visual constraints are added directly to this new space, while the traditional DBSCAN is revisited to capitalize on the availability of constraints and the reduced dimensional space. We modify DBSCAN with two new characteristics for story clustering: 1) constraint-based centroid selection and 2) adaptive radius. Experiments on TRECVID-2004 corpus demonstrate that CCC with visual constraints is more capable of mining news topics of varying densities, shapes and sizes, compared with traditional -means, DBSCAN, and spectral co-clustering algorithms.Index Terms-Multimedia topic detection and tracking, near-duplicate visual constraint, news story clustering, video data mining.

Effective Multimodality Fusion Framework for Cross-Media Topic Detection

“…The effectiveness of multi-modality fusion has been extensively demonstrated in various tasks [21]- [23], such as topic detection [2]- [4], [7], [21], [23], [24], multimedia event detection [25]- [27], video story summarization [28]- [30], and video annotation [31].…”

“…Many excellent works [38], [40]- [43] have been proposed to effectively detect near duplicate keyframe (NDK), which is widely used in the task of topic detection [3], [7], [23], [29], [44]. Hsu et al [44]utilized NDK and other multimodal features to track known topics across broadcasting news videos.…”

“…Shao et al [7] integrated NDK with other multi-modality features by a Star-structured K-partite Graph (SKG) to improve the topic detection performance. Wu et al [23] presented a weighted bipartite graph to discover topicrelated stories, where pairs of NDK are used as visual feature nodes to serve as visual constraints. T. Chen et al [3] fused the dense bursty tag groups with NDK to efficiently detect topics from web videos.…”

Video scene detection by link-constrained affinity-propagation

“…Scene Detection NDOs between shots provide a strong constraint to identify whether the shots belong to the same scene. In [9], Wu et al proposed to use near-duplicate keyframes (NDKs) to bridge news stories from various video sources over time. They treated such visual constraints as a "must-link" condition, i.e.…”

“…During clustering, we do not apply hard constraints, such as "must-link" [9], to force shots containing similar objects into the same group because the constraints may contain outliers and noises when performing NDO detection. We also relax the time constraint that a scene must consist of temporally connected shots.…”