Nudging Science Towards Fairer Evaluations: Evidence From Peer Review

Smirnova, Inna; Teplitskiy, Misha

doi:10.2139/ssrn.4190623

Cited by 7 publications

(9 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, a separating equilibrium might emerge where getting an anonymous manuscript is the same as receiving an unknown author's manuscript. Despite this concern, in a field experiment, voluntary anonymization helped increase the acceptance rate of unknown authors slightly (Smirnova et al, 2023), though not enough to offset the large biases estimated in the above field experiments. Still, if voluntary anonymization can help, then mandatory anonymization should help even more.…”

Section: Potential Solutionsmentioning

confidence: 90%

“…In finance, double-blind peer-review is standard at the top 3 journals, in economics singleblind peer-review is standard at the top 5 journals, reflecting a surprising difference in otherwise similar disciplines. 9 The costs of implementing it are small, 10 and it works partly even today in decreasing discrimination (Smirnova et al, 2023). It is hard to see why double-blind reviewing is not the standard in economics, as it is in other disciplines.…”

Section: Potential Solutionsmentioning

confidence: 99%

See 1 more Smart Citation

Economics Peer-Review: Problems, Recent Developments, and Reform Proposals

Siemroth

2023

SSRN Journal

View full text Add to dashboard Cite

Section: Potential Solutionsmentioning

confidence: 90%

Section: Potential Solutionsmentioning

confidence: 99%

Economics Peer-Review: Problems, Recent Developments, and Reform Proposals

Siemroth

2023

SSRN Journal

View full text Add to dashboard Cite

“…An alternative policy that lies between non-anonymization and compulsory anonymization is that of not making anonymization compulsory, but allowing the authors the option to anonymize their papers. The paper [31] studies such a policy in a number of journals from IOP publishers, finding that authors of 22% of papers volunteered to anonymize, and furthermore, the highest-prestige authors anonymized less often but still substantially. This policy increased the acceptance rate of papers by "low-prestige" authors by 5.6%.…”

Section: Related Workmentioning

confidence: 99%

The role of author identities in peer review

Shah

2023

PLoS ONE

View full text Add to dashboard Cite

There is widespread debate on whether to anonymize author identities in peer review. The key argument for anonymization is to mitigate bias, whereas arguments against anonymization posit various uses of author identities in the review process. The Innovations in Theoretical Computer Science (ITCS) 2023 conference adopted a middle ground by initially anonymizing the author identities from reviewers, revealing them after the reviewer had submitted their initial reviews, and allowing the reviewer to change their review subsequently. We present an analysis of the reviews pertaining to the identification and use of author identities. Our key findings are: (I) A majority of reviewers self-report not knowing and being unable to guess the authors’ identities for the papers they were reviewing. (II) After the initial submission of reviews, 7.1% of reviews changed their overall merit score and 3.8% changed their self-reported reviewer expertise. (III) There is a very weak and statistically insignificant correlation of the rank of authors’ affiliations with the change in overall merit; there is a weak but statistically significant correlation with respect to change in reviewer expertise. We also conducted an anonymous survey to obtain opinions from reviewers and authors. The main findings from the 200 survey responses are: (i) A vast majority of participants favor anonymizing author identities in some form. (ii) The “middle-ground” initiative of ITCS 2023 was appreciated. (iii) Detecting conflicts of interest is a challenge that needs to be addressed if author identities are anonymized. Overall, these findings support anonymization of author identities in some form (e.g., as was done in ITCS 2023), as long as there is a robust and efficient way to check conflicts of interest.

show abstract

“…hiding authors' identities from reviewers. Anonymization has been shown to reduce other forms of bias in peer review (Blank, 1991;Fox et al, 2023;Fox & Paine, 2019;Harris et al, 2017;Kowal et al, 2022;Smirnova et al, 2022). To test for the effect of anonymization on homophily we leverage a policy change among IOP journals that enabled authors to select into double-anonymous peer review (Smirnova et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

“…Anonymization has been shown to reduce other forms of bias in peer review (Blank, 1991;Fox et al, 2023;Fox & Paine, 2019;Harris et al, 2017;Kowal et al, 2022;Smirnova et al, 2022). To test for the effect of anonymization on homophily we leverage a policy change among IOP journals that enabled authors to select into double-anonymous peer review (Smirnova et al, 2022). The rollout of the anonymization policy was staggered across journals in as-if random order, making the policy an attractive instrumental variable for anonymization.…”

Section: Introductionmentioning

confidence: 99%

The effect of reviewer geographical diversity on evaluations is reduced by anonymizing submissions

Zumel Dumlao,

Teplitskiy

2023

Preprint

View full text Add to dashboard Cite

Scientists from low- and middle-income countries tend to have lower acceptance rates in academic journals than those from wealthier ones. This disparity may be partly caused by a lack of geographical diversity among peer reviewers, although rigorous evidence is lacking. We used administrative data on 205K submissions to 60 single-anonymous journals published by the Institute of Physics Publishing to measure whether reviewers favor work from their countries and whether authors from wealthier countries are more likely to be assigned to same-country reviewers (SCRs). We found that SCRs were 4.78 percentage points more likely to give a positive review compared to non-SCRs on the same manuscript. Authors from wealthier countries were up to twice as likely to be assigned to SCRs. In a sample of submissions that underwent double-anonymous review, the SCR homophily estimate was smaller and statistically similar to 0, suggesting that the homophily observed when identities are visible is an identity-based bias. Overall, authors from wealthy countries enjoy a structural advantage – they are more likely to be assigned to SCRs and receive their more favorable reviews. Diversifying the reviewer pool geographically would make acceptance rates across countries slightly more equal, likely by spreading out bias more evenly.

show abstract

Nudging Science Towards Fairer Evaluations: Evidence From Peer Review

Cited by 7 publications

References 33 publications

Economics Peer-Review: Problems, Recent Developments, and Reform Proposals

Economics Peer-Review: Problems, Recent Developments, and Reform Proposals

The role of author identities in peer review

The effect of reviewer geographical diversity on evaluations is reduced by anonymizing submissions

Contact Info

Product

Resources

About