Evaluating detection of near duplicate video segments

Abstract

The automatic detection of near duplicate video segments, such as multiple takes of a scene or different news video clips showing the same event, has received growing research interest in recent years. However, there is no agreed way of evaluating near duplicate detection algorithms. This makes it very hard to compare the performance of different algorithms, even if they are applied to the same data set. In this paper we have implemented several evaluation measures found in literature and we apply them to real algorithm outputs and a simulated result data set. We then calculate the correlation between the results obtained with the different measures in order to investigate whether they can be compared or not. The results show that the correlation between the measures is some cases quite low, and some measures are especially sensitive to certain types of deviations from the ground truth. However, a group of precision/recall type measures and two others are clearly correlated, though with moderate correlation coefficients. We also analyze the correlation between these measures and the subjective human judgment of the number of repeated segments in summary videos.

Content-Based Video Copy Detection Benchmarking at TRECVID

“…(Although NDCR via its parameterization supports a variety of application scenarios, comparison of results across studies beyond TRECVID would require comparable parameter settings [Bailer 2010]. ) In the first (pilot) year of the CCD task, a single application scenario was tested with the parameters of…”

A Survey of Advanced Content Management Tools for TV Postproduction

“…Di↵erent measures for evaluating near duplicate video detection are found in literature. A recent study [6] found the following measures are applied for evaluating near duplicate video detection: three variants of precision/recall based measures, a measure based on normalized mutual information, and three measures coming from classic copy detection, i.e., two measures used in the MUSCLE VCD benchmark [64] and the one used in the TRECVID benchmark [100] content-based copy detection task. The measure di↵ers in several dimensions, such as providing frame or segment precision, requiring the same segmentation for ground truth and results and support for evaluation of clustering per scene/topic.…”

“…In [6] the correlations of the di↵erent measures on real algorithm results as well as on simulated results have been analyzed, as well as their correlation with human judgment of redundancy from the TRECVID rushes experiments. The results show that depending on the type of di↵erences (segments added, dropped, shifted) between ground truth and results the correlation between the measures can in some cases be quite low, so that results obtained from di↵erent measures cannot be compared as it is sometimes found in literature.…”