A probabilistic ranking framework using unobservable binary events for video search

Abstract

Recent content-based video retrieval systems combine output of concept detectors (also known as high-level features) with text obtained through automatic speech recognition. This paper concerns the problem of search using the noisy concept detector output only. Unlike term occurrence in text documents, the event of the occurrence of an audiovisual concept is only indirectly observable. We develop a probabilistic ranking framework for unobservable binary events to search in videos, called PR-FUBE. The framework explicitly models the probability of relevance of a video shot through the presence and absence of concepts. From our framework, we derive a ranking formula and show its relationship to previously proposed formulas. We evaluate our framework against two other retrieval approaches using the TRECVID 2005 and 2007 datasets. Especially using large numbers of concepts in retrieval results in good performance. We attribute the observed robustness against the noise introduced by less related concepts to the effective combination of concept presence and absence in our method. The experiments show that an accurate estimate for the probability of occurrence of a particular concept in relevant shots is crucial to obtain effective retrieval results.

Reusing annotation labor for concept selection

“…Recently, Aly et al presented in [1] PRFUBE, a probabilistic retrieval method for the uncertain occurrence of concepts in shots, which estimates the probability of relevance, given a shot description of concept occurrences. In contrast to the Entropy method it considers both, the case in which the concept occurs and in which it does not.…”

“…We name our estimation method "score-based". Now, a set of concepts can be selected and a Concept Based IR method, such as PRFUBE [1] or Entropy [19], can use these values to execute the third step of the retrieval procedure -to search on the actual search collection.…”

“…For the Concept Based runs we use the recently proposed retrieval models PRFUBE [1] and Entropy [19]. For a baseline we choose the method from Hauff et al [5] where the Text IR scores for concept descriptions of single concepts are linearly scaled to form the probability P (C|R).…”

“…For a baseline we choose the method from Hauff et al [5] where the Text IR scores for concept descriptions of single concepts are linearly scaled to form the probability P (C|R). The parameters for the scaling are set according to [1] where the best setting was 0.05 for the lowest possible probability and a range of 0.6 for the TRECVID 2005 collection. For the TRECVID 2007 collection we used a range of 0.20.…”

“…However, regardless of the detection technique every concept based retrieval system needs to answer the following questions: (1) What are good concepts for an information need? (2) Can they be effectively detected?…”