The evaluation of citation distributions

Abstract

This paper reviews a number of recent contributions that demonstrate that a blend of welfare economics and statistical analysis is useful in the evaluation of the citations received by scientific papers in the periodical literature. The paper begins by clarifying the role of citation analysis in the evaluation of research. Next, a summary of results about the citation distributions' basic features at different aggregation levels is offered. These results indicate that citation distributions share the same broad shape, are highly skewed, and are often crowned by a power law. In light of this evidence, a novel methodology for the evaluation of research units is illustrated by comparing the high-and low-citation impact achieved by the US, the European Union, and the rest of the world in 22 scientific fields. However, contrary to recent claims, it is shown that mean normalization at the sub-field level does not lead to a universal distribution. Nevertheless, among other topics subject to ongoing research, it appears that this lack of universality does not preclude sensible normalization procedures to compare the citation impact of articles in different scientific fields.

Testing for universality of Mendeley readership distributions

“…Citation distributions have very different characteristics across sub-fields, a mean significantly higher than the median, and a limited share of articles accounting for a large share of citations (Albarrán, Crespo, Ortuño, & Ruiz-Castillo, 2011;Bornmann & Leydesdorff, 2017). Ruiz-Castillo (2012) confirms that citation distributions share the same broad shape and are highly skewed, adding that they are often crowned by a power law. For Franceschet (2011), in computer science, the skewness registered for conference publications is also more pronounced than that for journal papers.…”

Identifying potential “breakthrough” publications using refined citation analyses: Three related explorative approaches

“…Waltman & Schreiber, 2013). For methods 2a and 2b, we use the so-called Characteristics Scores and Scales (CSS) method suggested by Schubert, Glänzel and Braun (1987) and demonstrated in other bibliometric approaches by Ruiz-Castillo (2012). The interesting possibilities of using CSS for research performance analyses are discussed by Glänzel, Thijs and Debackere (2013).…”

Research assessment based on infrequent achievements: A comparison of the United States and Europe in terms of highly cited papers and Nobel Prizes

“…This graphical method was devised to solve and present data relevant to the research evaluation objectives addressed in this study with maximum clarity, but not to study the mathematical relationship between number of citations and frequency of papers. This relationship may be described by a power‐law function (Clauset, Shalizi, & Newman, ; Redner, ; Ruiz‐Castillo, ); this study uses this function without attempting to discuss a power‐law relationship versus other possible functions. It uses the empirical linear relationship in the lg‐lg plots only to predict the position of points of the function that could not be reliably determined by counting papers, so the citation range in which the putative power‐law tail starts and the reported values for the percentage of papers included in this tail are only approximations.…”