Citation gaming induced by bibliometric evaluation: A country-level comparative analysis

Abstract

It is several years since national research evaluation systems around the globe started making use of quantitative indicators to measure the performance of researchers. Nevertheless, the effects on these systems on the behavior of the evaluated researchers are still largely unknown. For investigating this topic, we propose a new inwardness indicator able to gauge the degree of scientific self-referentiality of a country. Inwardness is defined as the proportion of citations coming from the country over the total number of citations gathered by the country. A comparative analysis of the trends for the G10 countries in the years 2000-2016 reveals a net increase of the Italian inwardness. Italy became, both globally and for a large majority of the research fields, the country with the highest inwardness and the lowest rate of international collaborations. The change in the Italian trend occurs in the years following the introduction in 2011 of national regulations in which key passages of professional careers are governed by bibliometric indicators. A most likely explanation of the peculiar Italian trend is a generalized strategic use of citations in the Italian scientific community, both in the form of strategic author self-citations and of citation clubs. We argue that the Italian case offers crucial insights on the constitutive effects of evaluation systems. As such, it could become a paradigmatic case in the debate about the use of indicators in science-policy contexts.

How much is too much? The difference between research influence and self-citation excess

“…Today, both excessive self-citation and citation cartels are real concerns and have been documented (Ioannidis 2015;Fister et al 2016;Heneberg 2016;Fong and Wilhite 2017;Zaggl 2017;Scarpa et al 2018;Baccini et al 2019;Biagioli et al 2019;Seeber et al 2019;Biagioli and Lippman 2020). Excessive and artificial forms of self-citation have been discussed and studied for:…”

“…Citations are widely considered to be an indicator of a published work's significance but how can an analyst confirm whether a high citation count for an individual is a genuine reflection mark of influence or a consequence of other factors such as extraordinary, even excessive, self-referencing? The question is pertinent because self-citation has featured in recent publications that address the possible misrepresentation of research performance by individuals and, specifically, attempts to game citation scores (Baccini et al 2019;D'Antuono and Ciavarella 2019;Kacem et al 2019;Peroni et al 2019;Seeber et al 2019).…”

“…• Journals, especially to game the impact factor (Rousseau 1999;Frandsen 2007;Yu and Wang 2007;Campanario 2011;Wilhite and Fong 2012;Yu et al 2014;Chorus and Waltman 2016;Heneberg 2016;Yang et al 2016;Campanario 2018;Yu et al 2018;Copiello 2019;Ioannidis and Thombs 2019). • Institutions Thijs and Glänzel 2006;Schubert et al 2006;Hendrix 2009;Costas et al 2010;Gul et al 2017) • Nations (Aksnes 2003;Glänzel and Thijs 2004a;Glänzel et al 2006;Schubert et al 2006;Minasny et al 2010;Ladle et al 2012;Adams 2013;Tang et al 2015;Bakare and Lewison 2017;Bornmann et al 2018;Scarpa et al 2018;Baccini et al 2019;D'Antuono and Ciavarella 2019;Peroni et al 2019;Shehatta and Al-Rubaish 2019). Scarpa and Baccini and colleagues have reported that the Italian national research assessment exercise (VQR, Research Quality Evaluation), initiated at the beginning of the past decade, provoked strategic behavior among researchers, especially those early in their careers, that resulted in significantly higher rates of self-citation than previously seen (Scarpa et al 2018;Baccini et al 2019;Peroni et al 2019;Seeber et al 2019), although not all agree with this finding (D'Antuono and Ciavarella 2019).…”

The practice of self-citations: a longitudinal study

“…Also, the self-citation increment seemed not to relate with an increase in the international collaboration, that remained quite stable in Italy in the years considered in the analysis, which overlap with the time window we used in our study. Indeed, since 2010, Italy has become "the European country with the lowest international collaboration and the highest inwardness" (Baccini et al 2019).…”

“…see (Marzolla 2015(Marzolla , 2016 . In particular, Baccini, De Nicolao and Petrovich (2019) showed that the recent increase of self-citations recorded at the national level, looking at the affiliations of the scholars, seemed to be a peculiar trend in Italy. The novelty in our analysis is to consider author self-citations and author network self-citations on a broad set of scientific disciplines (24 in total, according to the classification provided by Scimago Journal Rank) by looking at the articles authored by the participants to the Italian Scientific Habilitation.…”