Investigating Assessment Standards in the Netherlands, Italy, and the United Kingdom

Abstract

The following contribution asks which role standards for research information play in practices of responsible research evaluation. The authors develop the notion of assessment standards against the background of functional standard classifications. The development of semantic and procedural assessment standards in the national research evaluation exercises of the Netherlands, Great Britain, and Italy are investigated using a qualitative case study design. A central finding of the study is that assessment standards incorporate conflicting values. A continuous tradeoff between the transparency of evaluation procedures and provided information as well as the variety of research outputs is being counterbalanced in all countries by compensating a higher level of semantic standardization with lower degrees of procedural standardization.

Shaping the Qualities, Values and Standards of Science. How Reporting Guidelines Improve the Transparency of Biomedical Research

“…PRISMA and its incorporation into a framework such as the Hong Kong Principles promises a more accountable form of research evaluation. Scholars have mentioned how research policymakers and funding bodies become more inclined toward standardized forms of research evaluation and actively support the development of various frameworks (Mejlgaard et al, 2020 ; Petersohn et al, 2020 ). Especially the standards' claim to properly capture aspects such as credibility, rigor or transparency serves as their promise to incentivize the right trajectories for science, rather than just any (Langfeldt et al, 2019 ; Peterson and Panofsky, 2020 ).…”

“…Originating from philosophical discussions about the proper epistemic values for science, scholars soonly placed transparency among other quality criteria such as plausibility, reliability or credibility and created what sometimes has been called "a tapestry of values" (Elliott, 2017) or value "portfolio" (Mårtensson et al, 2016). Such a tapestry not only visualizes the complexity of the aims and goals in science, but also how there can be multiple and even conflicting notions of research quality at the same time (Petersohn et al, 2020).…”

“…As such, peer review becomes more transparent because journals publish review reports and editorial decisions (Hartstein and Blümel, 2021 ; Waltman et al, 2022 ). In general, standardized forms of research assessment, e.g., ex ante assessments of grants and job applications (Hammarfelt and Rushforth, 2017 ), or ex post evaluations in peer review or bibliometric assessment, promise transparency by making evaluative procedures traceable and comprehensible (Petersohn et al, 2020 ). Combined under the label of “Open Science,” scholars hope that making various aspects of the research process more transparent increases public trust and accountability, leading to a “credibility revolution” (Vazire, 2018 , p. 411; see also Fecher and Friesike, 2014 ).…”

A comparison of systematic reviews and guideline-based systematic reviews in medical studies

“…In contemporary science policies, much emphasis is put on the evaluation of the producers of biomedical knowledge. The most recent trend in science evaluation, the incorporation of seemingly objective tools to measure publication output or citation impact gave rise to the idea that the former represents productivity, while the latter indicates quality, professionalism or excellence in biomedical research, (de Rijcke et al, 2016;Jappe, 2020;Petersohn et al, 2020). Biomedical researchers and other scientists have reacted to such incentives, not only by dividing their work into more publishable units, but also by in incorporating such evaluative categories deeply into their epistemic practices (Müller & de Rijcke, 2017).…”