Evaluating audio skimming and frame rate acceleration for summarizing BBC rushes

Abstract

For the first time in 2007, TRECVID considered structured evaluation of automated video summarization, utilizing BBC rushes video. In 2007, we conducted user evaluations with the published TRECVID summary assessment procedure to rate a cluster method for producing summaries, a 25x (sampling every 25th frame), and pz (emphasizing pans and zooms). Data from 4 human assessors shows significant differences between the cluster, pz, and 25x approaches. The best coverage (text inclusion performance) is obtained by 25x, but at the expense of 25x taking the most time to evaluate and judged as being the most redundant. Method pz was easier to use than cluster and rated best on redundancy. A question following the TRECVID workshop was whether simple speed-ups would still work at 50x or 100x, leading to a study with 15 human assessors looking at pzA (pz but with better audio), 25x, 50x, and 100x summaries (these latter 3 with an unsynchronized more comprehensive audio track as well). 100x gives the fastest time on task but with poor usability and performance. PzA gives the best usability measures but poor time on task and performance. 25x does well on performance as before, with 50x doing just as well but with much less time on task and better ease of use and redundancy scores. Based on these results, 50x with its audio skimming is recommended as the best way to summarize video rushes materials.

Exploring the utility of fast-forward surrogates for bbc rushes

“…We extract three different features to construct our junk frame detectors: HSV color histogram features, SIFT features and speech recognition features. Color histogram features provide solid performance to detect black frames, white frames, and color bar frames, as witnessed in the TRECVID 2007 summarization runs [2,5]. However, the main challenge is to detect clapper frames.…”

“…Before any work was begun on TRECVID 2008, we conducted an empirical study with 15 human assessors on the TRECVID 2007 rushes summarization task [2], set up specifically to look at the utility of 25x, 50x, and 100x fast-forward surrogates for the BBC rushes. By simply sampling every 25th frame, you create a 4% video summary, which we label 25x in agreement with [10].…”

“…The audio is incomprehensible at 25x playback, but some of the BBC rushes dialogue seemed to hold value based on casual inspection of the development data. So, we augmented the 25x, 50x, and 100x video with regular speed narration to produce the tested summaries [2]. For the visual component of the Nx summaries, every Nth frame was selected with no consideration given for noise-filtering.…”

“…The accelerated 2% summary provided excellent performance equivalent to 25x, but with dramatically faster time on task, and no significant drop in the ease of use or redundancy satisfaction metrics. For the tested material, 50x is recommended, with 100x dropping off significantly in both performance and rated usability [2]. When the invitation to participate came out for the TRECVID 2008 rushes summarization task with the target summary size reduced from 4% (used in 2007) to 2%, this allowed for the 50x summary to be submitted as a video surrogate having an established empirical record of success [2,10].…”

“…For the tested material, 50x is recommended, with 100x dropping off significantly in both performance and rated usability [2]. When the invitation to participate came out for the TRECVID 2008 rushes summarization task with the target summary size reduced from 4% (used in 2007) to 2%, this allowed for the 50x summary to be submitted as a video surrogate having an established empirical record of success [2,10]. The next section discusses how we tweaked attributes of the 50x summary to investigate the relative contributions of other such attributes, using the common benchmark metrics discussed in [6].…”