A Stochastic Treatment of Learning to Rank Scoring Functions

Abstract

Learning to Rank, a central problem in information retrieval, is a class of machine learning algorithms that formulate ranking as an optimization task. The objective is to learn a function that produces an ordering of a set of documents in such a way that the utility of the entire ordered list is maximized. Learning-to-rank methods do so by learning a function that computes a score for each document in the set. A ranked list is then compiled by sorting documents according to their scores. While such a deterministic mapping of scores to permutations makes sense during inference where stability of ranked lists is required, we argue that its greedy nature during training leads to less robust models. This is particularly problematic when the loss function under optimization-in agreement with ranking metrics-largely penalizes incorrect rankings and does not take into account the distribution of raw scores. In this work, we present a stochastic framework where, instead of a deterministic derivation of permutations from raw scores, permutations are sampled from a distribution defined by raw scores. Our proposed sampling method is differentiable and works well with gradient descent optimizers. We analytically study our proposed method and demonstrate when and why it leads to model robustness. We also show empirically, through experiments on publicly available learning-to-rank datasets, that the application of our proposed method to a class of ranking loss functions leads to significant model quality improvements.

Evaluating Stochastic Rankings with Expected Exposure

“…Recently, Bruch et al [4] demonstrate that learning to rank models can be optimized towards expected values of relevance metrics computed over multiple rankings sampled based on estimated relevance. While not developed in the context of deploying a stochastic ranker, we adopt some of the methodologies therein in our experiments.…”

“…Unfortunately, this sampling process is non-differentiable and, therefore, prohibitive to a large class of models, including those that learn by gradient descent. We address this by adopting the method proposed by Bruch et al [4]. To construct a sampled ranking , we reparameterize the probability distribution by adding independently drawn noise samples from the Gumbel distribution [23] to y and sorting items by the "noisy" probability distributioñ ,˜(…”

Neural methods for effective, efficient, and exposure-aware information retrieval

“…It is noteworthy, that enumerating all distinct document permutations can be computationally challenging even for a moderately sized set of candidates. More recently, Bruch et al [418] demonstrated a mechanism for sampling rankings from the Plackett-Luce distribution using the reparameterization trick [395] that is amenable to gradient-based optimization. Their approach involves adding independently drawn noise samples from the Gumbel distribution [419] and then deriving the approximate rank of the document following the method proposed by Qin et al [420] and Wu et al [421].…”

Permutation Equivariant Document Interaction Network for Neural Learning to Rank

“…For our experiments, we use two public learning-to-rank datasets with numerical features, and two large-scale proprietary datasets. 3 We discard queries with no relevant documents, similar to evaluation in [4]. ick Access.…”

“…On the public datasets, we compare attn-DIN with the RankLib 4 and LightGBM [10] implementations of LambdaMART, and state-of-theart neural ranking algorithms: SetRank [14], Deep Listwise Context Model (DLCM) [1], Groupwise Scoring Functions (GSF) [2], and Feed-Forward Neural Network (with ReLU activations) with Gumbel Approximate NDCG loss [3]. We tune the hyperparameters of LightGBM and set both the number of iteration and the number of leaves to be 2,000 for WEB30K and 500 for Istella.…”