Rank Aggregation via Heterogeneous Thurstone Preference Models

Abstract

We propose the Heterogeneous Thurstone Model (HTM) for aggregating ranked data, which can take the accuracy levels of different users into account. By allowing different noise distributions, the proposed HTM model maintains the generality of Thurstone's original framework, and as such, also extends the Bradley-Terry-Luce (BTL) model for pairwise comparisons to heterogeneous populations of users. Under this framework, we also propose a rank aggregation algorithm based on alternating gradient descent to estimate the underlying item scores and accuracy levels of different users simultaneously from noisy pairwise comparisons. We theoretically prove that the proposed algorithm converges linearly up to a statistical error which matches that of the state-of-the-art method for the single-user BTL model. We evaluate the proposed HTM model and algorithm on both synthetic and real data, demonstrating that it outperforms existing methods.

Adaptive Sampling for Heterogeneous Rank Aggregation from Noisy Pairwise Comparisons

“…We conduct thorough experiments and show that the proposed algorithm can perform as good as the oracle algorithm and is significantly more sample efficient than all baseline algorithms. One immediate and interesting future direction may be to extend our adaptive sampling algorithm to more complicated models such as the heterogeneous Bradley-Terry-Luce model and the heterogeneous Thurstone Case V model (Jin et al, 2020).…”

“…In our experiment, we use a similar setup as that of Jin et al (2020). In particular, we consider a set of users [M ], whose accuracies are set by p u (i, j) = (1 + exp(γ u (s j − s i ))) −1 , for u ∈ [M ] and any items i, j ∈ [N ], where parameter γ u is used as an scaling factor of the user accuracy and s i , s j are the utility scores of the corresponding items in the BTL model.…”

“…To rank a set of items from noisy pairwise comparisons or preferences is a widely studied topic in machine learning (Braverman and Mossel, 2008;Weng and Lin, 2011;Ren et al, 2019;Jin et al, 2020). This is also referred to as rank aggregation.…”

Lagrangian Inference for Ranking Problems

“…There are other general frameworks on ranking problems such as the Thurstone model (Thurstone 1927, Vojnovic and Yun 2017, Orbán-Mihálykó et al 2019, Jin et al 2020 and Plackett-Luce model (Guiver andSnelson 2009, Hajek et al 2014). For instance, Jin et al (2020) propose a heterogeneous…”

“…We point out that the BTL model is invariant that if we multiply ω * i , or increase θ * , by a constant c, the distribution of y Specific examples of f include 1 θ = 0, θ 1 = 1 (θ 1 is the preference score of the first item), among others (Negahban et al 2017, Jin et al 2020).…”