Weak evidence of country- and institution-related status bias in the peer review of abstracts

Nielsen, Mathias Wullum; Baker, Christine; Brady, Emer; Petersen, Michael Bang; Andersen, Jens Peter

doi:10.7554/elife.64561

Cited by 8 publications

(2 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nonacademic experience, international experience, mobility or training may constitute important elements when assessing an individual's potential for future performance, especially in earlier stages of academic career, when other achievements may be still lacking. It also gives room to specific factors which may be affecting individuals' trajectories, such as institutional reputation (Morgan et al 2018;Nielsen et al 2021).…”

Section: Evaluative Dimensionsmentioning

confidence: 99%

Valuation regimes in academia: Researchers’ attitudes towards their diversity of activities and academic performance

Robinson-García

Costas

Nane

et al. 2023

Research Evaluation

View full text Add to dashboard Cite

Evaluation systems have been long criticized for abusing and misusing bibliometric indicators. This has created a culture by which academics are constantly exposing their daily work to the standards they are expected to perform. In this study, we investigate whether researchers’ own values and expectations are in line with the expectations of the evaluation system. We conduct a multiple case study of five departments in two Dutch universities to examine how they balance between their own valuation regimes and the evaluation schemes. For this, we combine curriculum analysis with a series of semi-structured interviews. We propose a model to study the diversity of academic activities and apply it to the multiple case study to understand how such diversity is shaped by discipline and career stage. We conclude that the observed misalignment is not only resulting from an abuse of metrics but also by a lack of tools to evaluate performance in a contextualized and adaptable way.

show abstract

Section: Evaluative Dimensionsmentioning

confidence: 99%

Valuation regimes in academia: Researchers’ attitudes towards their diversity of activities and academic performance

Robinson-García

Costas

Nane

et al. 2023

Research Evaluation

View full text Add to dashboard Cite

show abstract

“…They are deeply intertwined with race and tend to favor White PIs over others. If redaction reduced halo effects, it would suggest that blinded review models might improve fairness [47,48]. On the other hand, it may have been that when PI identity was deleted, the scientific narrative lost context and was consequently degraded.…”

Section: Why?mentioning

confidence: 99%

An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes

Nakamura

Mann

Lindner

et al. 2021

eLife

View full text Add to dashboard Cite

Background: Blinding reviewers to applicant identity has been proposed to reduce bias in peer review. Methods: This experimental test used 1200 NIH grant applications, 400 from Black investigators, 400 matched applications from White investigators, and 400 randomly selected applications from White investigators. Applications were reviewed by mail in standard and redacted formats. Results: Redaction reduced, but did not eliminate, reviewers' ability to correctly guess features of identity. The primary, pre-registered analysis hypothesized a differential effect of redaction according to investigator race in the matched applications. A set of secondary analyses (not pre-registered) used the randomly selected applications from White scientists and tested the same interaction. Both analyses revealed similar effects: Standard format applications from White investigators scored better than those from Black investigators. Redaction cut the size of the difference by about half (e.g. from a Cohen's d of 0.20 to 0.10 in matched applications); redaction caused applications from White scientists to score worse but had no effect on scores for Black applications. Conclusions: Grant-writing considerations and halo effects are discussed as competing explanations for this pattern. The findings support further evaluation of peer review models that diminish the influence of applicant identity. Funding: Funding was provided by the NIH.

show abstract

Impact of author characteristics on outcomes of single- versus double-blind peer review: a systematic review of comparative studies in scientific abstracts and publications

Giannakakos,

Karanfilian,

Dimopoulos

et al. 2024

Scientometrics

View full text Add to dashboard Cite

The purpose of this systematic review was to assess the role of double-blind (DB) peer review on bias against authors when compared to single-blind (SB) peer review in scientific publications. Using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, a search of databases including Pubmed, Embase, and Web of Science, without language or date restrictions, was conducted to identify original research studies comparing peer-review outcomes between DB and SB methods based on any of the following author characteristics: gender, race, geographic location, personal prestige, institutional prestige. Studies were ranked Level I, II, or III for quality of evidence based on a modified version of the Oxford Center for Evidence-Based Medicine 2011 Levels of Evidence. Of 29 studies included, five level I studies, the highest quality evidence, showed that in SB peer review, the following author characteristics were associated with more positive outcomes: male gender, White race, location of the US or North America, well-published or known in their field, or affiliation with prestigious institutions. The evidence of whether DB peer review resulted in better outcomes for authors lacking these characteristics was more discordant, possibly due to lack of effective blinding or due to unblinded editor decisions. However, if bias reduction is defined as elimination of advantages afforded to only certain types of authors, DB peer review should be considered.

show abstract

Weak evidence of country- and institution-related status bias in the peer review of abstracts

Cited by 8 publications

References 34 publications

Valuation regimes in academia: Researchers’ attitudes towards their diversity of activities and academic performance

Valuation regimes in academia: Researchers’ attitudes towards their diversity of activities and academic performance

An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes

Impact of author characteristics on outcomes of single- versus double-blind peer review: a systematic review of comparative studies in scientific abstracts and publications

Contact Info

Product

Resources

About