Gender and other potential biases in peer review: Analysis of 38,250 external peer review reports

Abstract

Background The Swiss National Science Foundation (SNSF) supports fundamental and use-inspired research in all disciplines. Peer reviewers assess the proposals submitted to the SNSF. We examined whether the gender of applicants and reviewers and other factors influenced the summary scores awarded. Methods We analysed 38,250 reports on 12,294 grant applications across all disciplines 2006 to 2016. Proposals were rated on a scale from 1 (=worst) to 6 (=best) by 26,836 reviewers. We used linear mixed effects regression models adjusted for research topic, applicant's age, nationality, affiliation and calendar period to examine associations, and interactions between gender of the applicant and other variables. Results In univariable analysis, male applicants received more favourable evaluation scores than female applicants (+0.19 points; 95% CI 0.14-0.23), and male reviewers awarded higher scores than female reviewers (+0.12; 95% CI 0.08-0.15). Applicant-nominated reviewers awarded higher scores than reviewers nominated by the SNSF (+0.53; 95% CI 0.50-0.56), and reviewers affiliated with research institutions outside of Switzerland more favourable scores than reviewers affiliated with Swiss institutions (+0.53; 95% CI 0.49-0.56). In multivariable analysis, differences between male and female applicants were attenuated (to +0.08; 95% CI 0.04-0.13) whereas results changed little for source of nomination and affiliation of reviewers. There was an interaction between gender of applicant and reviewer, and between gender of applicant and calendar period. Male reviewers gave higher scores than female reviewers, with a greater difference for male than for female applicants (P=0.037 from test of interaction). The gender difference increased after September 2011, when new evaluation forms were introduced (P=0.033 from test of interaction). Conclusions Our study showed that peer review of grant applications at SNSF might be prone to biases stemming from different applicant and reviewer characteristics. The SNSF abandoned the nomination of peer reviewers by applicants, and made members of panels aware of the other systematic differences in scores. The new form introduced in 2011 may inadvertently have given more emphasis to the applicant’s track record, and a revision is now under discussion. We encourage other funders to conduct similar studies, in order to improve the evidence base for rational and fair research funding.

Community review: a robust and scalable selection system for resource allocation within open science and innovation communities

“…In extreme cases, the grant review program can be more costly than simply allocating small grants to each applicant, as in the case of the NSERC grant system of 2008 (3). In addition, the allocation of grants has shown to suffer from various biases, such as the composition of the grant panel (4), gender and geographical location (5), group based dynamics personality triumphing over other qualitative factors (6)(7)(8), and socio-psychological factors such as group dynamics and personality traits triumphing over other qualitative factors (8,9). Overall, selection results are only weakly predictive of future performance (10).…”

The troubles with peer review for allocating research funding

“…The Swiss National Science Foundation (SNSF) found that reviewers nominated by applicants tended to score their applications higher than reviewers selected by the Foundation. It concluded that their peer‐review mechanism might be subject to bias and stopped requiring reviewer proposals by applicants [preprint: ].…”

Editorial: Women in chemical engineering

“…Women, especially those from underrepresented groups such as Black, Indigenous, and people of color, bear higher burdens of service-related activities in what is commonly known as a "minority tax" (Rodríguez et al, 2021;Williamson et al, 2021). Conversely, in metrics that are traditionally rewarded in academia, such as prestigious international awards (Meho, 2021) and peerreviewed grants and publications (Day et al, 2020;Severin et al, 2020), women are often disadvantaged. For example, a cross-sectional analysis of peer review reports from the Swiss National Science Foundation revealed that grants submitted by male applicants scored more favorably than those submitted by female applicants (Severin et al, 2020).…”