Quantitative research assessment: using metrics against gamed metrics

Abstract

Quantitative bibliometric indicators are widely used and widely misused for research assessments. Some metrics have acquired major importance in shaping and rewarding the careers of millions of scientists. Given their perceived prestige, they may be widely gamed in the current “publish or perish” or “get cited or perish” environment. This review examines several gaming practices, including authorship-based, citation-based, editorial-based, and journal-based gaming as well as gaming with outright fabrication. Different patterns are discussed, including massive authorship of papers without meriting credit (gift authorship), team work with over-attribution of authorship to too many people (salami slicing of credit), massive self-citations, citation farms, H-index gaming, journalistic (editorial) nepotism, journal impact factor gaming, paper mills and spurious content papers, and spurious massive publications for studies with demanding designs. For all of those gaming practices, quantitative metrics and analyses may be able to help in their detection and in placing them into perspective. A portfolio of quantitative metrics may also include indicators of best research practices (e.g., data sharing, code sharing, protocol registration, and replications) and poor research practices (e.g., signs of image manipulation). Rigorous, reproducible, transparent quantitative metrics that also inform about gaming may strengthen the legacy and practices of quantitative appraisals of scientific work.

The Replication Dilemma: Potential Challenges in Measuring Replication Value—A Commentary on Isager, Van’t Veer, & Lakens (2024)

“…However, in the current ecosystem, citation counts are not an unbiased measure of impact. Therefore, as part of the metric, citation counts should be taken into after screening for fraudulent highly cited scientists, or by using metrics against gamed metrics (Ioannidis & Maniadis, 2024).…”

Insight in article retractions inBritish Journal of Pharmacologybetween the years 2000–2023. Implications for reducing fraud‐containing articles

“…A 2012 paper that investigated retractions of drug therapy studies found that 72% of the retractions were due to research misconduct (Samp et al, 2012). Evaluation of scientists based on their publications and citations for career advancement is one of the incentives for the publication of fraudcontaining articles (Ioannidis & Maniadis, 2024). Interestingly, a number of authors have been associated with more than one retraction (Kuroki & Ukawa, 2018;Samp et al, 2012) and this has been designated as "serial cases of fraud", by Samp et al (2012).…”

Evolving patterns of extreme publishing behavior across science

“…Regardless of the exact of mix of genuinely high output, spurious authorship standards, and outright unethical research practices (Ioannidis & Maniadis, 2024), extreme publishing authors appear to enjoy high success in terms in citation impact, especially when raw citation counts are considered. 44% of the most-cited authors across science in terms of raw citations are extreme publishing authors.…”

“…However, this may be a bad idea as it may further exaggerate publication bias and other selection biases. Instead, it may be more realistic and appropriate to monitor extreme publishing behaviors in centralized, standardized databases (Ioannidis & Maniadis, 2023), as we have done here. This monitoring should allow careful in-depth assessments of extreme patterns for single authors, teams, institutions, and countries.…”