Comparison of rank aggregation methods based on inherent ability

Abstract

Ranking is a common task for selecting and evaluating alternatives. In the past few decades, combining rankings results from various sources into a consensus ranking has become an increasingly active research topic. In this study, we focus on the evaluation of rank aggregation methods. We first develop an experimental data generation method, which can provide ground truth ranking for alternatives based on their “inherent ability.” This experimental data generation method can generate the required individual synthetic rankings with adjustable accuracy and length. We propose characterizing the effectiveness of rank aggregation methods by calculating the Kendall tau distance between the aggregated ranking and the ground truth ranking. We then compare four classical rank aggregation methods and present some useful findings on the relative performances of the four methods. The results reveal that both the accuracy and length of individual rankings have a remarkable effect on the comparison results between rank aggregation methods. Our methods and results may be helpful to both researchers and decision‐makers.

The rank boost by inconsistency in university rankings: Evidence from 14 rankings of Chinese universities

“…While different aggregation algorithms all claim to be superior to existing ones when proposed, the baseline algorithms and the testing samples are all different from case to case. Although there are some reviews or comparisons of aggregation methods (Brancotte et al, 2015;Li, Wang, & Xiao, 2019;Xiao, Deng et al, 2017), most of them cover only a few algorithms and the conclusions may not be general enough. Indeed, it was unclear which method is most appropriate to aggregate a small number of long lists that are incomplete, uneven, and with ties.…”

Rank‐based analysis method to determine OWA weights and its application in group decision making

“…In decision making problems, usually there are several criteria for evaluating options, for example, courses of action, candidates and the like. There are two main kernels in a multicriteria decision making theory, namely determination method of weights to different criteria and aggregation method with these weights . Up to present, a large quantity of research has been developed to the two kernels including a predominant method called the OWA operator.…”

“…There are two main kernels in a multicriteria decision making theory, namely determination method of weights to different criteria and aggregation method with these weights. [2][3][4][5] Up to present, a large quantity of research has been developed to the two kernels including a predominant method called the OWA operator.…”

Comparing Rank Aggregation Methods based on Mallows Model

“…For example, results from peer review of research proposals and articles, an essential element in R&D process and the academic community worldwide, can be combined to achieve better set of candidate proposals and help improve quality of decision output [8]. Although there are works [9][10][11][12] comparing rank aggregation methods from different aspects, all of them didn't have a suitable, consistent and general data generation mechanism. As far as we know, [9] is the first one to consider data generation model to produce sets of permutations with various degree of consensus, with focus on balance between search time and algorithm performance, but they didn't consider other list types.…”

“…Furthermore, they all considered traditional indices to evaluate and compare the performance of rank aggregation methods and didn't realize that those indices are problematic more or less, which will be discussed in section 4. [10] developed a data generation model which can generate the required synthetic base rankers with adjustable accuracy and length and found both the accuracy and length have a remarkable effect on the comparison results between rank aggregation methods. However, their model is not controllable to some extent, such as the number of ties, and their performance index is also not applicable since ground truth ranking is not easy to find and it is exactly what we are pursuing.…”