Model-based evaluation of scientific impact indicators

Abstract

Using bibliometric data artificially generated through a model of citation dynamics calibrated on empirical data, we compare several indicators for the scientific impact of individual researchers. The use of such a controlled setup has the advantage of avoiding the biases present in real databases, and it allows us to assess which aspects of the model dynamics and which traits of individual researchers a particular indicator actually reflects. We find that the simple average citation count of the authored papers performs well in capturing the intrinsic scientific ability of researchers, regardless of the length of their career. On the other hand, when productivity complements ability in the evaluation process, the notorious h and g indices reveal their potential, yet their normalized variants do not always yield a fair comparison between researchers at different career stages. Notably, the use of logarithmic units for citation counts allows us to build simple indicators with performance equal to that of h and g. Our analysis may provide useful hints for a proper use of bibliometric indicators. Additionally, our framework can be extended by including other aspects of the scientific production process and citation dynamics, with the potential to become a standard tool for the assessment of impact metrics.

Ranking in evolving complex networks

“…However, it is long known that the self-reinforcing preferential attachment process strongly contributes to the evolution of paper citation count [153,154,243] which suggests that one should approach high citation counts with some caution. Indeed, a recent study shows that the aggregation of logarithmic citation counts leads to better identification of able authors than the aggregation of original citation counts [244]. The little reliability of a single highly-cited paper was one of the reasons that motivated the introduction of the popular h-index that aims to estimate the productivity and research impact of authors.…”

“…Besides machine-learning approaches being traditionally strong when enough data are available, the method is not built on any specific model of paper popularity growth as it directly uses empirical paper popularity profiles. This can be an important advantage, given that the network growth model used in [73,244] certainly lacks some features of the real citation network. For example, Petersen et al [252] considered the impact of author reputation on the citation dynamics and found that the author's reputation dominates the paper citation growth rate of little cited papers.…”

“…This strategy opens different alternative possibilities. One can use models of network growth [71,244] where the nodes are endowed with some fitness parameter, grow synthetic networks of a given size and, a posteriori, evaluate the metrics' ability to uncover the nodes' latent fitness [79,244] (see also section III). Another possibility is to use real data and test the metrics' ability to unearth the intrinsic spreading ability of the nodes according to some model of diffusion on the network.…”

Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data

“…Citation-based metrics for impact build on the premise that the number of citations received by a scientific paper (or a patent) is a reliable proxy for its scientific (or technological) impact. Such metrics are used not only to assess the impact of individual papers, but also to evaluate the overall research output of research units such as individual researchers (Hirsch, 2005;Radicchi et al, 2009;Medo and Cimini, 2016), research institutes (Charlton and Andras, 2007;West et al, 2013), and journals (Harzing and Van Der Wal, 2009;González-Pereira et al, 2010), for example. The relative ease with which new metrics of research impact can be designed has contributed to their proliferation (Mingers and Leydesdorff, 2015;Waltman, 2016;Todeschini and Baccini, 2016), and uncritical use of such metrics has eventually met a strong opposition (Hicks et al, 2015;Rijcke et al, 2016;Leydesdorff et al, 2018).…”

The long-term impact of ranking algorithms in growing networks

“…The main message of our work is that network-based growth models can help us not only to understand the impact of network growth mechanisms on the rankings by a given algorithm [22,26,29], but also to estimate the impact of the adoption of different ranking algorithms by a given system. In other words, we can investigate not only how the past evolution of the system influenced the current rankings, but also how the adopted rankings may influence the future evolution of the system.…”