Evaluating scientific research: Belief, hindsight bias, ethics, and research evaluation

Hom, Harry L.; Nuland, Abigail L. Van

doi:10.1002/acp.3519

Cited by 5 publications

(1 citation statement)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This partly explains the variation in formal assessment criteria across disciplinary contexts, career stages and history (Mantai and Marrone 2023). Moreover, decisions are also made by groups and individuals, and objectivity in decision-making is hence at risk from the social (Bornmann, Mutz, and Daniel 2007;Wennerås and Wold 1997;Ginther et al 2011;Teplitskiy et al 2018), political (Wennerås and Wold 1997;Altbach, Yudkevich, and Rumbley 2015;Torrance 2016) and cognitive (Hom Jr. and Van Nuland 2019;East 2016;Juárez Ramos 2019) biases and preferences of evaluators. Furthermore, although reliance on quantitative indicators may have the veil of objectivity, they too are fundamentally affected by biases, either those baked into the metrics (Strathern 2000;Islam and Greenwood 2022), or those of the assessors using the metrics (Hammarfelt and Rushforth 2017).…”

Section: Introductionmentioning

confidence: 99%

Understanding the social and political dimensions of research(er) assessment: Interpretative flexibility and hidden criteria

Ross-Hellauer¹,

Bonn²,

Horbach³

2023

Preprint

View full text Add to dashboard Cite

Debates about appropriate, fair and effective ways of assessing research and researchers have raged through the scientific community for decades, recently mostly concerned with discussing the merits and limitations of metric-based, quantitative assessments versus peer review-based, qualitative alternatives. Ample attention has been paid to formal assessment criteria, building to a consensus that less emphasis should be placed on quantification, while supporting open and diverse sets of criteria. Yet the theory and evidence upon which such policy reform depends is still surprisingly sparse. Based on qualitative free-text responses from 121 respondents gathered during an international survey of active researchers, this study examines how criteria are applied in practice and how those being assessed perceive informal criteria to determine the outcomes of assessments. While confirming the general critique on over-emphasising quantification, respondents particularly identify a mismatch between formal criteria and actual evaluation practices. Hidden criteria, including social, political, and demographic factors, are perceived important, especially in intransparent assessment procedures, opening up for assessors’ interpretative flexibility. This adds to ongoing discussions on the performativity of assessment criteria and lays bare a tension between the rigidity and flexibility of criteria and the extent to which these can be transparently communicated.

show abstract

Section: Introductionmentioning

confidence: 99%

Understanding the social and political dimensions of research(er) assessment: Interpretative flexibility and hidden criteria

Ross-Hellauer¹,

Bonn²,

Horbach³

2023

Preprint

View full text Add to dashboard Cite

show abstract

Facilitating open science practices for research syntheses: PreregRS guides preregistration

Schneider

Backfisch

Lachner

2022

Research Synthesis Methods

View full text Add to dashboard Cite

Researchers increasingly engage in adopting open science practices in the field of research syntheses, such as preregistration. Preregistration is a central open science practice in empirical research to enhance transparency in the research process and it gains steady adoption in the context of conducting research synthesis. From an interdisciplinary perspective, frameworks and particularly templates are lacking which support researchers preparing a preregistration. To this end, we introduce preregRS, a template to guide researchers across disciplines through the process of preregistering research syntheses. We utilized an R Markdown template file to provide a framework that structures the process of preparing a preregistration. Researchers can write up the preregistration using the template file similar to filling out a form, with the template providing additional hints and further information for the decisions along the framework. We integrated the R Markdown template in an R package for easy installation and use, but also provide a browser‐based option for users granting low‐barrier access. PreregRS constitutes a first step to facilitate and support preregistration with research syntheses for all disciplines. It further adds to establishing open science practices in conducting research syntheses.

show abstract

Retrospective and prospective hindsight bias: Replications and extensions of Fischhoff (1975) and Slovic and Fischhoff (1977)

Chen

Kwan

Yeung

et al. 2021

Journal of Experimental Social Psychology

View full text Add to dashboard Cite

Evaluating scientific research: Belief, hindsight bias, ethics, and research evaluation

Cited by 5 publications

References 42 publications

Understanding the social and political dimensions of research(er) assessment: Interpretative flexibility and hidden criteria

Understanding the social and political dimensions of research(er) assessment: Interpretative flexibility and hidden criteria

Facilitating open science practices for research syntheses: PreregRS guides preregistration

Retrospective and prospective hindsight bias: Replications and extensions of Fischhoff (1975) and Slovic and Fischhoff (1977)

Contact Info

Product

Resources

About