Fisher kernel based relevance feedback for multimodal video retrieval

Abstract

This paper proposes a novel approach to relevance feedback based on the Fisher Kernel representation in the context of multimodal video retrieval. The Fisher Kernel representation describes a set of features as the derivative with respect to the log-likelihood of the generative probability distribution that models the feature distribution. In the context of relevance feedback, instead of learning the generative probability distribution over all features of the data, we learn it only over the top retrieved results. Hence during relevance feedback we create a new Fisher Kernel representation based on the most relevant examples. In addition, we propose to use the Fisher Kernel to capture temporal information by cutting up a video in smaller segments, extract a feature vector from each segment, and represent the resulting feature set using the Fisher Kernel representation. We evaluate our method on the MediaEval 2012 Video Genre Tagging Task, a large dataset, which contains 26 categories in 15.000 videos totalling up to 2.000 hours of footage. Results show that our method significantly improves results over existing state-of-the-art relevance feedback techniques. Furthermore, we show significant improvements by using the Fisher Kernel to capture temporal information, and we demonstrate that Fisher kernels are well suited for this task.

Zero-Example Event Search using MultiModal Pseudo Relevance Feedback

“…The idea is to learn a ranking function by optimizing the pair-wise or list-wise orders between pseudo positive and negative samples. In [19], the relevance judgment over the top-ranked videos is provided by users. Then an SVM is trained using visual features represented in the Fisher vector.…”

Daily Living Activities Recognition via Efficient High and Low Level Cues Combination and Fisher Kernel Representation

“…Other works employed temporal rules with high-level concepts [14]. To the best of our knowledge the only work that used Fisher Kernel to model the temporal variation in videos is [18]. They employed a frame-based global feature descriptor for a movie-genre classification scenario.…”

“…Each of the 18 points in Fig.1(a) is the centroid of the bounding box of the corresponding body part of size L (as shown in Fig.1(b)). The accuracy of the body pose estimation is computed by comparing the positions of the groundtruth bounding box B GT i and of the estimated bounding box B E i , for each body part i = 1, ..., 18. If the overlap of B E i with B GT i is more than 80%, the body part is considered as being correctly detected.…”

Fisher Kernel Temporal Variation-based Relevance Feedback for video retrieval

“…This paper extends our previous work [3,4] by including evaluation on a new video dataset, evaluating more feature extraction schemes, analyzing the influence of multiple relevance feedback iterations, and including a computational complexity analysis. To summarize, our main contributions are as follows:…”