On the Use of Visual Soft Semantics for Video Temporal Decomposition to Scenes

Abstract

Abstract-This work examines the possibility of exploiting, for the purpose of video segmentation to scenes, semantic information coming from the analysis of the visual modality. This information, in contrast to the low-level visual features typically used in previous approaches, is obtained by application of trained visual concept detectors such as those developed and evaluated as part of the TRECVID High-Level Feature Extraction Task. A large number of non-binary detectors is used for defining a high-dimensional semantic space. In this space, each shot is represented by the vector of detector confidence scores, and the similarity of two shots is evaluated by defining an appropriate shot semantic similarity measure. Evaluation of the proposed approach is performed on two test datasets, using baseline concept detectors trained on a dataset completely different from the test ones. The results show that the use of such semantic information, which we term "visual soft semantics", contributes to improved video decomposition to scenes.

Concept-based Image Clustering and Summarization of Event-related Image Collections

“…When working with low-level features, the squared Euclidean or the Euclidean distance, depending on the clustering algorithm, are typically used in the literature. However, for the model vectors we use an alternative distance measure which was introduced in [17], and was shown to be appropriate for confidence scores comparison. According to it, if C(Ii) and C(I k ) are two model vectors for the images Ii and I k respectively, the distance D of C(Ii) and C(I k ) is defined as:…”

Improving Interactive Video Retrieval by Exploiting Automatically-Extracted Video Structural Semantics

“…However, manual processing of large collections of video for extracting structural semantics is practically infeasible, and the state-of-the-art techniques for performing this task automatically generate results that still deviate considerably from perfection (e.g. [9], [10]). Therefore, it is by no means straightforward to say that video structural semantics extracted automatically by current stateof-the-art techniques are useful in interactive retrieval, nor is it of course possible to quantify their potential contribution without detailed experimentation.…”

“…[10], [12], further exploit higher-level information such as visual concept and audio event detection results in order to come to a more accurate extraction of the videos' structural semantics. Specifically, in [10] the possibility of exploiting, for the purpose of video segmentation to scenes, semantic information coming from the analysis of the visual modality, was examined.…”

“…For the purpose of the study presented in this work, 6 different variations of the method of [10] were used (Table II). These differ in the information they use as input for extracting the video structural semantics (i.e., lowlevel visual features only for variations M1 to M3; lowlevel features and the responses of visual concept detectors ("visual soft semantics") for variations M4 to M6) and in the setting of their parameters (i.e., the shot similarity threshold for variations M1 to M3; the number of considered concept detectors and the strategy for their selection for variations M4 to M6).…”

“…These differ in the information they use as input for extracting the video structural semantics (i.e., lowlevel visual features only for variations M1 to M3; lowlevel features and the responses of visual concept detectors ("visual soft semantics") for variations M4 to M6) and in the setting of their parameters (i.e., the shot similarity threshold for variations M1 to M3; the number of considered concept detectors and the strategy for their selection for variations M4 to M6). In general, the variations of the scene segmentation algorithm that take into account visual soft semantics were shown in [10] to produce scene segmentation results that are in better agreement with ground truth scene boundaries. The interested reader is referred to the aforementioned work for further details on these algorithms.…”