Professional standards in bibliometric research evaluation? A meta-evaluation of European assessment practice 2005–2019

Abstract

Despite growing demand for practicable methods of research evaluation, the use of bibliometric indicators remains controversial. This paper examines performance assessment practice in Europe-first, identifying the most commonly used bibliometric methods and, second, identifying the actors who have defined widespread practices. The framework of this investigation is Abbott's theory of professions, and I argue that indicator-based research assessment constitutes a potential jurisdiction for both individual experts and expert organizations. This investigation was conducted using a search methodology that yielded 138 evaluation studies from 21 EU countries, covering the period 2005 to 2019. Structured content analysis revealed the following findings: (1) Bibliometric research assessment is most frequently performed in the Nordic countries, the Netherlands, Italy, and the United Kingdom. (2) The Web of Science (WoS) is the dominant database used for public research assessment in Europe. (3) Expert organizations invest in the improvement of WoS citation data, and set technical standards with regards to data quality. (4) Citation impact is most frequently assessed with reference to international scientific fields. (5) The WoS classification of science fields retained its function as a de facto reference standard for research performance assessment. A detailed comparison of assessment practices between five dedicated organizations and other individual bibliometric experts suggests that corporate ownership and limited access to the most widely used citation databases have had a restraining effect on the development and diffusion of professional bibliometric methods during this period.

Research Performance Assessment Issues: The Case of Kazakhstan

“…Despite the inconsistency of the h-index and the inappropriateness of the journal impact factor for research performance assessment (Waltman & van Eck, 2012;van Leeuwen, 2012), the bibliometric approach to research assessment has become popular. It is most frequently performed in the Nordic countries, the Netherlands, Italy and the United Kingdom (Jappe, 2020). Bibliometrics are considered more attractive than peer review because they demand much less time and effort.…”

Interpreting Bibliometric Data

“…However, because the academic sector, at discipline level, has taken little or no responsibility for understanding and interpreting quantitative indicators based on citation data, de-facto and generic standards of research excellence have been defined at system level by others (including scientometricians and data providers) without being challenged by the implied authority of the domain experts. While the possible forms of analysis are diverse, Jappe (2020) reviewed 138 evaluation studies from 21 EU countries, covering the period 2005 to 2019, and found that bibliometric research assessment, which was common to the United Kingdom, the Nordic network, the Netherlands and Italy, was most frequently based on ‘citation impact’ metrics, usually with reference to international scientific fields.…”

“…The most widely used standard indicator for ‘citation impact’ is the number of citations received by a publication, normalized “with reference to international scientific fields” ( Jappe, 2020 ). It is generally understood that papers with higher citation counts are associated with greater influence or ‘impact’ since they reflect acknowledgment by other researchers ( Garfield, 1955 ).…”

Bibliometric Reports for Institutions: Best Practices in a Responsible Metrics Scenario

“…Local or national bibliometric products should be used to complement areas that are not well covered by international databases, as occurs in the arts and social sciences. According to Jappe (2020) , only one out of every four bibliometric assessment studies uses national sources. Current Research Information Systems (CRIS), as well as institutional administrative databases and other non-bibliometric sources, can offer a more precise picture of research in the institution and are critical to offer accurate and significant results.…”