Comparison of performance of rank aggregation algorithms in aggregating a small number of long rank lists

Abstract

Rank aggregation aims to combine multiple rank lists into a single one, which has wide applications in recommender systems, link prediction, metasearch, proposal selection, and so on. Some existing studies have summarized and compared different rank aggregation algorithms. However, most of them cover only a few algorithms, the data used to test algorithms do not have a clear statistical property, and the metric used to quantify the aggregated results has certain limitations. Moreover, different algorithms all claim to be superior to existing ones when proposed, the baseline algorithms, the testing samples, and the application scenario are all different from case to case. Therefore, it is still unclear which algorithm is better for a particular task. Here we review nine rank aggregation algorithms and compare their performances in aggregating a small number of long rank lists. We assume an algorithm to generate different types of rank lists with known statistical properties and cause a more reliable metric to quantify the aggregation results. We find that despite the simplicity of heuristic algorithms, they work pretty well when the rank lists are full and have high similarities. In some cases, they can reach or even surpass the optimization-based algorithms in performance. The number of ties in the list will reduce the quality of the consensus rank and increase fluctuations. The quality of aggregated rank changes non-monotonically with the number of rank lists that need to be combined. Overall, the algorithm FAST outperforms all others in three different rank types, which can sufficiently complete the task of aggregating a small number of long rank lists.

The rank boost by inconsistency in university rankings: Evidence from 14 rankings of Chinese universities

“…In a recently study, we performed a comprehensive test on nine rank aggregation algorithms. We introduced a variation of Mallows model (Irurozki, Calvo et al, 2016) to generate synthetic ranking lists whose physical property is known and tuned for different circumstances (Chen, Zhu et al, 2020). The synthetic ranking lists provide us the ground truth for comparisons, where we find that the branch and bound algorithm FAST by Amodio et al (2016) is most appropriate in our task.…”

A novel similarity measure for mining missing links in long-path networks

“…Recently, network embedding techniques that are instances of representation learning on networks have been widely applied in link prediction. [42][43][44][45] Embedding-based predictors are derived from network embedding techniques, which attempt to automate feature engineering by projecting nodes in a network into a relatively low-dimensional latent space, to locally preserve node's neighborhoods. In this study, after representing nodes in a network as vectors, we apply t-distributed stochastic neighbor embedding (t-SNE) algorithm [46] to reduce the vector dimensions.…”