Rocchio-Based Relevance Feedback in Video Event Retrieval

Abstract

This paper investigates methods for user and pseudo relevance feedback in video event retrieval. Existing feedback methods achieve strong performance but adjust the ranking based on few individual examples. We propose a relevance feedback algorithm (ARF) derived from the Rocchio method, which is a theoretically founded algorithm in textual retrieval. ARF updates the weights in the ranking function based on the centroids of the relevant and non-relevant examples. Additionally, relevance feedback algorithms are often only evaluated by a single feedback mode (user feedback or pseudo feedback). Hence, a minor contribution of this paper is to evaluate feedback algorithms using a larger number of feedback modes. Our experiments use TRECVID Multimedia Event Detection collections. We show that ARF performs significantly better in terms of Mean Average Precision, robustness, subjective user evaluation, and run time compared to the state-of-the-art.

Impact of Interaction Strategies on User Relevance Feedback

“…Some evaluation protocols use this for labeling the suggestions as positive or negative [6,29]. Other evaluation protocols, especially those that work with large-scale collections, also add additional arbitrary negatives [16,23,25,36]. Analytic Quality uses artificial actors which solve an analytic task derived from an existing benchmark/user task, measuring precision and recall over time and estimating the user's insight gain [38].…”

“…While the plethora of work on automated evaluations contributes to show the effectiveness of URF systems in various fields, the evaluation methodology only captures the behavior of a specific interaction strategy which may not be a strategy a real user will resort to. Aside from this there has also been work that has focused on evaluating systems with real users [23,26,29,39]. URF systems typically showcase up to 30 images in each round and depending on the restrictions they can label as many items as they want [29], be limited to label a few examples [26], or only label examples as positive [25].…”

“…Traditionally, evaluation of URF focuses on measuring the classification performance of the model used to retrieve items [11,29,34,35]. Recently, automated evaluation protocols have been used to evaluate URF systems with a focus on artificial users [16,23,36] for large multimedia collections. However, these are based on one interaction strategy where the artificial user labels relevant items as positives and the system labels everything else as negatives.…”