Evaluating paper and author ranking algorithms using impact and contribution awards

Dunaiski, Marcel; Visser, Willem; Geldenhuys, Jaco

doi:10.1016/j.joi.2016.01.010

Cited by 47 publications

(28 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This leads us to the conclusion that rescaled PageRank is the best-performing metric overall. With respect to previous works [18,21,29,42] that claimed the superiority of network-based metrics in identifying important papers, our results clarify the essential role of paper age in determining the metrics' performance: rescaled PageRank excels and PageRank performs poorly in identifying MLs short after their publication, and the performance of the two methods becomes comparable only 15 years after the MLs are published. Qualitatively similar results are found for an alternative list of APS outstanding papers which only includes works that have led to Nobel prize for some of the authors (the list is provided in the Table S1).…”

Section: Introductionsupporting

confidence: 75%

Identification of milestone papers through time-balanced network centrality

Mariani¹,

Medo²,

Zhang³

2016

Journal of Informetrics

View full text Add to dashboard Cite

Citations between scientific papers and related bibliometric indices, such as the h-index for authors and the impact factor for journals, are being increasingly used -often in controversial ways -as quantitative tools for research evaluation. Yet, a fundamental research question remains still open: to which extent do quantitative metrics capture the significance of scientific works? We analyze the network of citations among the 449, 935 papers published by the American Physical Society (APS) journals between 1893 and 2009, and focus on the comparison of metrics built on the citation count with network-based metrics. We contrast five article-level metrics with respect to the rankings that they assign to a set of fundamental papers, called Milestone Letters, carefully selected by the APS editors for "making long-lived contributions to physics, either by announcing significant discoveries, or by initiating new areas of research". A new metric, which combines PageRank centrality with the explicit requirement that paper score is not biased by paper age, is the best-performing metric overall in identifying the Milestone Letters. The lack of time bias in the new metric makes it also possible to use it to compare papers of different age on the same scale. We find that networkbased metrics identify the Milestone Letters better than metrics based on the citation count, which suggests that the structure of the citation network contains information that can be used to improve the ranking of scientific publications. The methods and results presented here are relevant for all evolving systems where network centrality metrics are applied, for example the World Wide Web and online social networks. An interactive Web platform where it is possible to view the ranking of the APS papers by rescaled PageRank is available at the address http://www.sciencenow.info.

show abstract

Section: Introductionsupporting

confidence: 75%

Identification of milestone papers through time-balanced network centrality

Mariani¹,

Medo²,

Zhang³

2016

Journal of Informetrics

View full text Add to dashboard Cite

show abstract

“…While we have analyzed in detail the presence of age and field bias in the ranking, it still remains to evaluate the actual ranking performance of the newly proposed indicators in artificial data [32] or in real data where the ground truth is provided by some external source [18,33]. Another important issue is the comparison between metrics based on citation count and metrics that take the whole citation network into account to determine papers' score.…”

Section: Discussionmentioning

confidence: 99%

Quantifying and suppressing ranking bias in a large citation network

Vaccario

Medo

Wider

2017

Journal of Informetrics

View full text Add to dashboard Cite

It is widely recognized that citation counts for papers from different fields cannot be directly compared because different scientific fields adopt different citation practices. Citation counts are also strongly biased by paper age since older papers had more time to attract citations. Various procedures aim at suppressing these biases and give rise to new normalized indicators, such as the relative citation count. We use a large citation dataset from Microsoft Academic Graph and a new statistical framework based on the Mahalanobis distance to show that the rankings by well known indicators, including the relative citation count and Google's PageRank score, are significantly biased by paper field and age. We propose a general normalization procedure motivated by the z-score which produces much less biased rankings when applied to citation count and PageRank score.

show abstract

“…For example, network centrality metrics can be evaluated according to their ability to identify expert-selected significant nodes. Benchmarks of this kind include (but are not limited to) identification of expert-selected movies [16,245], identification of awarded conference papers [301,302] or of editor-selected milestone papers [165], identification of researchers awarded with international prizes [13,303].…”

Section: Perspectives and Conclusionmentioning

confidence: 99%

Ranking in evolving complex networks

et al. 2017

View full text Add to dashboard Cite

Complex networks have emerged as a simple yet powerful framework to represent and analyze a wide range of complex systems. The problem of ranking the nodes and the edges in complex networks is critical for a broad range of real-world problems because it affects how we access online information and products, how success and talent are evaluated in human activities, and how scarce resources are allocated by companies and policymakers, among others. This calls for a deep understanding of how existing ranking algorithms perform, and which are their possible biases that may impair their effectiveness. Well-established ranking algorithms (such as the popular Google's PageRank) are static in nature and, as a consequence, they exhibit important shortcomings when applied to real networks that rapidly evolve in time. The recent advances in the understanding and modeling of evolving networks have enabled the development of a wide and diverse range of ranking algorithms that take the temporal dimension into account. The aim of this review is to survey the existing ranking algorithms, both static and time-aware, and their applications to evolving networks. We emphasize both the impact of network evolution on well-established static algorithms and the benefits from including the temporal dimension for tasks such as prediction of real network traffic, prediction of future links, and identification of highly-significant nodes.Comment: 54 pages, 16 figure

show abstract

Evaluating paper and author ranking algorithms using impact and contribution awards

Cited by 47 publications

References 13 publications

Identification of milestone papers through time-balanced network centrality

Identification of milestone papers through time-balanced network centrality

Quantifying and suppressing ranking bias in a large citation network

Ranking in evolving complex networks

Contact Info

Product

Resources

About