Gaming the Metrics

Abstract

Traditionally, the assumption has been that academic misconduct emerges primarily in response to "publish or perish" pressures. Robert Slutsky, a UC San Diego cardiologist famously caught in 1986 reporting imaginary experiments, was, at one point, putting out one article every ten days (Lock and Wells, 2001). "Publish or perish," however, is no longer the sole incentive for misconduct. New practices are emerging that are not limited to the production of fraudulent publications but are aimed instead at enhancing, often in unethical or fraudulent ways, the evaluation of their importance or "impact" (Biagioli, 2016). "Publish or perish" is merging with "impact or perish." 1 This is related to but different from the predictable gaming of academic performance indicators one would expect from Goodhart's law: as soon as an indicator becomes a target, gaming ensues, which forecloses its ability to function as a good indicator. 2 That may take the form, for instance, of massaging the definition of what counts as a "successful student" in metrics about schools' performance, or of what counts as a "peer-reviewed" paper in faculty evaluation protocols. It could also involve aligning one's practices to metrics-relevant parameters, like capping classes' enrollment to nineteen students to have them fit the US News and World Report's definition of "small class," which is rewarded in its ranking of universities. But we now find authors and editors who move beyond this kind of gaming to create (rather than tweak) metricenhancing evidence, such as citations to one's work or to the work published in a given journal so as to boost its impact factor. We argue that the growing reliance on institutional metrics of evaluation does not just provide incentives for these kinds of manipulations, but also creates their conditions of possibility. They would not have come into being were it not for the new metrics-based "audit culture" of academia (

The SCOPE framework – implementing ideals of responsible research assessment

“…A mantra often used by the INORMS REG is that ‘where there is a prize there is a game’. This refers to the fact that where there is a lot at stake in a particular evaluation (reputationally and financially), the entities being evaluated will naturally be incentivised to alter their behaviours in ways that enable them to perform well ( Biagioli & Lippman, 2020 ). In some cases, this is indeed the purpose of an evaluation: evaluators will seek to assess a particular dimension (e.g., open research) to incentivise it.…”

Choices of immediate open access and the relationship to journal ranking and publish-and-read deals

“…Similarly, in Europe, 75% of 186 universities in a European University Association survey report, applied impact factor to evaluate researchers' output (Saenen et al, 2019 ). Publishing in high ranking journals has led to the introduction of monetary rewards (Quan et al, 2017 ), which means that journals literally can be regarded as “mints,” with the impact factor being a journal's actual value (Biagioli and Lippman, 2020 ).…”

Periódicos científicos com indexação descontinuada: a Coleção SciELO Brasil

“…Ao abordar a interface entre periódicos científicos e indexadores, a pesquisa se insere nos estudos sobre comunicação científica da ciência da informação. Periódicos científicos são um dos objetos de estudo mais investigados na área, especialmente pela condição de ser a parte mais visível de uma pesquisa e unidade de avaliação de desempenho de pesquisadores e instituições (Gingras, 2020). Krauskopf (2018) analisou periódicos com indexação descontinuada na Scopus, verificando que os sites oficiais dos títulos ainda mencionavam a indexação na base.…”