Field-normalized citation impact indicators using algorithmically constructed classification systems of science

Abstract

We study the problem of normalizing citation impact indicators for differences in citation practices across scientific fields. Normalization of citation impact indicators is usually done based on a field classification system. In practice, the Web of Science journal subject categories are often used for this purpose. However, many of these subject categories have a quite broad scope and are not sufficiently homogeneous in terms of citation practices. As an alternative, we propose to work with algorithmically constructed classification systems. We construct these classification systems by performing a large-scale clustering of publications based on their citation relations. In our analysis, 12 classification systems are constructed, each at a different granularity level. The number of fields in these systems ranges from 390 to 73,205 in granularity levels 1 to 12. This contrasts with the 236 subject categories in the WoS classification system. Based on an investigation of some key characteristics of the 12 classification systems, we argue that working with a few thousand fields may be an optimal choice. We then study the effect of the choice of a classification system on the citation impact of the 500 universities included in the 2013 edition of the CWTS Leiden Ranking. We consider both the MNCS and the PP top 10% indicator. Globally, for all the universities taken together citation impact indicators generally turn out to be relatively insensitive to the choice of a classification system. Nevertheless, for individual universities, we sometimes observe substantial differences between indicators normalized based on the journal subject categories and indicators normalized based on an appropriately chosen algorithmically constructed classification system. Acknowledgements. This is the second version of a Working Paper of the same title that appeared in this series in March 2014. This paper was conceived while Ruiz-Castillo enjoyed the hospitality of the Centre for Science and Technology Studies, Leiden University, The Netherlands, during the 2013 spring term. RuizCastillo also acknowledges financial help from the Spanish MEC through grant ECO2011-29762. 2

The Herrero-Villar approach to citation impact

“…Comparing alternative evaluation procedures usually involves two key elements: (1) the extent of re-rankings, and (2) the importance of cardinal differences between those evaluations (see inter alia Waltman et al, 2012, andRuiz-Castillo &. As a first approximation, the first aspect might be partially revealed by rank correlation coefficients, and also by the Ulam distance between rankings.…”

“…Yet we understand that the premium is interesting in this evaluation context. 10 Perianes-Rodriguez & Ruiz-Castillo (2015) find that a move from the alternative fractional approach to the multiplicative approach does not cause dramatic differences in co-authorship patterns and citation impact values. Nevertheless, ceteris paribus, the gainers with a move from the fractional to the multiplicative approach are characterized by (i) a low co-authorship rate for citation distributions as a whole, but a high co-authorship rate in the upper tail of these distributions; (ii) a low citation impact performance, and (iii) a small number of solo articles.…”

Comparative End-To-End Evaluation of Research Organizations

“…Of course at this stage, we assume that all publications are in the same discipline and from a coeval window so that normalization is not an issue. Normalization is only an additional detail that can be rationally worked out 5 . If C is thought of as a first-order indicator of performance, then it is possible to bring in the idea of an higherorder energy-like term X = iC = i 2 P, as another indicator of bibliometric performance.…”

Visualization and analysis of SCImago Journal & Country Rank structure via journal clustering

“…Although some authors claim that there is a lower accuracy of subject classifications and science maps when journals are used as the unit of analysis (Gómez et al, 1996;Klavans and Boyack, 2010;Ruiz-Castillo and Waltman, 2015), numerous researchers have successfully applied clustering algorithms on journal matrices and networks of citation, cocitation and/or coupling. Rafols and Leydesdorff (2009) carried out a comparative analysis of four classification systems: two generated by indexers -that is, content-based-and two by means of automatic clustering algorithms decomposing the aggregated JCR journal-journal citation matrix.…”