Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data

Abstract

Despite the increasing use of citation-based metrics for research evaluation purposes, we do not know yet which metrics best deliver on their promise to gauge the significance of a scientific paper or a patent. We assess 17 networkbased metrics by their ability to identify milestone papers and patents in three large citation datasets. We find that traditional information-retrieval evaluation metrics are strongly affected by the interplay between the age distribution of the milestone items and age biases of the evaluated metrics. Outcomes of these metrics are therefore not representative of the metrics' ranking ability. We argue in favor of a modified evaluation procedure that explicitly penalizes biased metrics and allows us to reveal metrics' performance patterns that are consistent across the datasets. PageRank and LeaderRank turn out to be the best-performing ranking metrics when their age bias is suppressed by a simple transformation of the scores that they produce, whereas other popular metrics, including citation count, HITS and Collective Influence, produce significantly worse ranking results.

Limits of PageRank-based ranking methods in sports data

“…If a ground truth set is available, a comparative assessment on real data is possible. This can be made more robust by considering multiple real datasets and multiple ground truth sets as done recently in (Xu et al, 2020) to compare ranking metrics for citation data. If a ground truth set is not available but a credible model for a given system exists, an assessment using synthetic data (as we have used here) is a practical alternative.…”

Characterizing research leadership on geographically weighted collaboration network

“…LeaderRank is further applied to identify the influential nodes in complex products and systems (Li et al, 2019); in power grids (Zhou et al, 2019); in manufacturing services (Wu et al, 2019). Notably, in the field of "Library & Information Science", Xu et al (2020) found that LeaderRank had the best performance in ranking science and technology citation data, compared with other 17 network-based metrics. However, it is noteworthy that all the previous variants of PageRank, including LeaderRank, only consider the topological features of nodes while ignoring other non-topological features, especially the spatial features that are very important to the academic performance and impact.…”

Network-based ranking in social systems: three challenges

“…This suggests that PageRank's temporal bias can be removed by rescaling the scores with a transformation that ensures that the average score of the nodes and its standard deviation are independent of node age [40]. When such a transformation is applied, the resulting 'rescaled' score can detect much earlier important nodes, with useful implications for the early detection of milestone papers [40,41], patents [41,42], and movies [43]. The benefit from this procedure is exemplified, again, by the paper that reported the first direct observation of gravitational waves [38]: the paper is ranked 16th by rescaled PageRank at the end of 2016, which constitutes a substantial improvement compared to the 12 482nd position by the original PageRank, and suggests that the paper deserves a place among the most significant ones in the APS corpus.…”

“…A key challenge is that the performance of an algorithm in one of these problems does not predict its performance in another one (cross-problem variability). For example, by comparing 17 network-based ranking algorithms, a recent study [41] found that time-rescaled versions of PageRank and its variant LeaderRank [11] are the best-performing algorithms in the identification of expert-selected seminal papers and patents. PageRank is also effective in identifying influential researchers [53].…”

“…In the problem of identifying expert-selected important nodes (papers or patents) in science and technology, the age distribution of the expert-selected nodes can significantly impact on the algorithms' performance: if the expert-selected nodes are old ones, performance evaluation metrics that ignore this bias will favor ranking algorithms that are biased in favor of old nodes [40,41,55]. 'Corrected' performance evaluation metrics that penalize biased metrics are not affected by this confounding effect [41]. However, there is not yet a unique and universally-agreed way to evaluate ranking algorithms for scientific and technological impact.…”