To intervene or not to intervene; is that the question? On the role of scientometrics in research evaluation

Abstract

Recent high-profile statements, criticisms, and boycotts organized against certain quantitative indicators (e.g., the DORA declaration) have brought misuses of performance metrics to the center of attention. A key concern captured in these movements is that the metrics appear to carry authority even where established agents of quality control have explicitly outlined limits to their validity and reliability as measurement tools. This raises a number of challenging questions for those readers of this journal who are implicated in questions of indicator "production" and, by extension, "effects." In this opinion piece we wish to critically engage the question of how producers of indicators can come to terms with their role as (partly) responsible parties in the current age of evaluative bibliometrics. We do so through the illuminating case of the professional scientometrics community.

Approaching Impact Meaningfully in Medical Education Research

“…Second, the responsibility for applying certain bibliometric indicators is spread across many stakeholders in the current "citation infrastructure" 286 including scientometricians, publishers, librarians, policymakers, evaluators, research managers, consultancies, researchers, and other users. 287 As a result, reforming systems of research assessment is a distributed and complex task. 288 An exploratory study in Dutch biomedicine 289 details the multiple ways that performance metrics affect knowledge-production processes.…”

Proceedings of the Conference "Impact of Social Sciences and Humanities for a European Research Agenda - Valuation of SSH in mission-oriented research" VIENNA 2018

“…We build on a previous work that problematises the claim to detachment, objectivity, and neutrality that has characterised dominant modes of research evaluation (Candea et al, 2015;Daston and Galison, 2007). Instead, we take seriously that the act of representing quality is also an intervention (De Rijcke and Rushforth, 2015).…”

A Simple Paradigm for Augmenting the Euclidean Index to Reflect Journal Impact and Visibility

“…Administrators often fail to appreciate the nuances of scholar-and journal-ranking mechanisms (for example, De Rijcke & Rushforth, 2015;Iglesias & Pecharromán, 2007), which are now mind-numbingly abundant (for example, Bornmann, Mutz, Hug, & Daniel, 2011). Anecdotal discussions with deans and provosts suggest that they often perceive scholarly output through an artificially discretized lens.…”