Structured learning of two-level dynamic rankings

Abstract

For ambiguous queries, conventional retrieval systems are bound by two conflicting goals. On the one hand, they should diversify and strive to present results for as many query intents as possible. On the other hand, they should provide depth for each intent by displaying more than a single result. Since both diversity and depth cannot be achieved simultaneously in the conventional static retrieval model, we propose a new dynamic ranking approach. In particular, our proposed two-level dynamic ranking model allows users to adapt the ranking through interaction, thus overcoming the constraints of presenting a one-size-fits-all static ranking. In this model, a user's interactions with the first-level ranking are used to infer this user's intent, so that second-level rankings can be inserted to provide more results relevant to this intent. Unlike previous dynamic ranking models, we provide an algorithm to efficiently compute dynamic rankings with provable approximation guarantees. We also propose the first principled algorithm for learning dynamic ranking functions from training data. In addition to the theoretical results, we provide empirical evidence demonstrating the gains in retrieval quality over conventional approaches.

Learning Socially Optimal Information Systems from Egoistic Users

“…We model φ F (x t , y t ) using a submodular aggregation of its components, which is a well accepted method for modeling diversity [9,14]. To simplify the exposition, we focus on rankings as objects y, but analogous constructions also work for other types of objects.…”

“…More generally, any monotone increasing and concave function of a∈A a can be used. It was shown [9,14] that this allows for a broad class of performance measures to be modeled, including many common IR performance metrics (e.g. NDCG, Precision, Coverage).…”

“…The dataset contains 17 queries, each with binary relevance judgments for 7 to 56 different user types, which we translate into binary utility values. Similar to previous work [9], we consider the probability of a user type to be proportional to the number of documents relevant to that user type. Also following [9], we only consider documents that are relevant to at least 1 user type to focus the experiments on learning to diversify, not learning to determine relevance.…”

“…Similar to previous work [9], we consider the probability of a user type to be proportional to the number of documents relevant to that user type. Also following [9], we only consider documents that are relevant to at least 1 user type to focus the experiments on learning to diversify, not learning to determine relevance. Table 1 summarizes some key properties of the data.…”

“…Some learning approaches exist as well and have been shown to outperform manually tuned methods [7][8][9][10]. Unfortunately, the practical use of those learning methods is limited, since most require expert annotated training data that explicitly lists all facets of an information need (e.g.…”