Semi-Supervised Rank Learning for Multimedia Known-Item Search

Abstract

Known Item Search (KIS) is a specialized task of the general multimedia search problem. It describes the scenario where a user has previously seen a video and wants to find it again in a large collection using a text description. While there exists only one correct answer to a query (or topic), the goal is to return a ranked list of videos most likely to satisfy the request. This search problem includes content from speech, visual, and meta-data, and it is not clear how the individual modalities should be combined in the final result. Reranking models have been shown to be effective in problems such as image search, but the single ground truth video for a topic presents a challenge for building a model. In this paper, we propose a semi-supervised rank learning approach to the multimedia problem. We use a large training set of topics and ground truth videos to learn a pairwise ranking model based on gradient boosted regression trees. We define a learning feature space that consists of features derived from topics, videos, and topic-video dependent results. To overcome the KIS class imbalance problem, a set of pseudo positive training examples are identified from each of the multimedia modalities. This semi-supervised approach uses a ground truth video to select similar videos in each of the individual modalities. We then model the similarities as a graph and use a K-Step Markov approach to estimate the importance of nodes in the graph relative to the truth root node.

aMM: Towards adaptive ranking of multi-modal documents

“…Each video and its ASR naturally form a multimodal document. We conducted experiments on the [6,46], we employed term frequency and inverse document frequency weighting scheme (tf-idf) to compute the initial ranking list for each given query, which has been shown as an effective approach for KIS task. Based on the initial ranking list, we implemented our rerank- ing model and its competitors.…”

“…In this work, we use modality to refer to the medium type and feature view to represent the feature type extracted from each modality. For example, in the known-item search (KIS) problem [6], each document consists of a video and its textual ASR (automatic speech recognition). Meanwhile, each of these two modalities can be represented by a rich set of feature views.…”