Reusing historical interaction data for faster online learning to rank for IR

Abstract

We summarize the findings from Hofmann et al. [6]. Online learning to rank for information retrieval (IR) holds promise for allowing the development of "self-learning" search engines that can automatically adjust to their users. With the large amount of e.g., click data that can be collected in web search settings, such techniques could enable highly scalable ranking optimization. However, feedback obtained from user interactions is noisy, and developing approaches that can learn from this feedback quickly and reliably is a major challenge. In this paper we investigate whether and how previously collected (historical) interaction data can be used to speed up learning in online learning to rank for IR. We devise the first two methods that can utilize historical data (1) to make feedback available during learning more reliable and (2) to preselect candidate ranking functions to be evaluated in interactions with users of the retrieval system. We evaluate both approaches on 9 learning to rank data sets and find that historical data can speed up learning, leading to substantially and significantly higher online performance. In particular, our preselection method proves highly effective at compensating for noise in user feedback. Our results show that historical data can be used to make online learning to rank for IR much more effective than previously possible, especially when feedback is noisy.

Probabilistic Multileave Gradient Descent

“…The advantage of P-MGD is that it can learn faster by having n, the number of candidates, in Algorithm 1 exceed the length of the result list. Candidate Preselection (CPS) [5], unlike MGD, does not alter the number of candidates compared per impression. It speeds up learning by reusing historical data to select…”

“…We show experimentally that P-MGD significantly outperforms state-of-the-art online learning to rank methods in terms of online performance, without sacrificing offline performance and at greater learning speed than those methods. In particular, we include comparisons between P-MGD on the one hand and multiple types of DBGD and multileaved gradient descent methods [14,MGD] and candidate preselection [5,CPS] on the other. We answer the following research questions: (RQ1) Does P-MGD convergence on a ranker of the same quality as MGD and CPS?…”

Multileave Gradient Descent for Fast Online Learning to Rank

“…DBGD implements a stochastic gradient descent method to find the best ranker in an infinite space of rankers. This algorithm has been extended before by Hofmann et al [11] such that it would reuse historical interaction data and not just live user interactions. Alternative methods are based on the k-armed bandit formulation by [32] and assume a finite space of possible rankers, the best of which needs to be found.…”

“…A major advantage of PI is that it can also infer preferences between rankers that were not originally interleaved. This allows one to learn from historical interaction data [11]. However, PI risks showing users poor rankings.…”

Online Learning to Rank for Information Retrieval

“…• More advanced methods that build on DBGD such as Probabilistic Multileave Gradient Decent [23,27] and DBGD with Candidate Preselection [12].…”