Potential bias in peer review of grant applications at the Swiss National Science Foundation

Abstract

Background: The Swiss National Science Foundation (SNSF) supports fundamental and use-inspired research in all academic disciplines. As part of the evaluation procedure, grant applications to the SNSF are reviewed by external reviewers. The legitimacy of funding decisions depends on its ability to base funding decisions solely on the scientific merit of grant applications. Aim: We examined whether the following factors influenced the scores given to grant applications submitted to the SNSF: (1) source of nomination of the reviewer, (2) the gender of the applicant and the reviewer, and (3) the country of affiliation of the reviewer. Methods and Data: We gathered data on 38,250 external reviews of 12,294 unique grant applications across all disciplines between 2006 and 2016. Proposals were rated on a scale from 1 (=poor) to 6 (=outstanding) by 26,836 reviewers. We used linear mixed effects regression models adjusted for research topic, applicant’s age, nationality and affiliation. Results: We found that in univariable analysis applicant-nominated reviewers awarded higher evaluation scores than reviewers nominated by the SNSF. Further, reviewers affiliated with research institutions outside of Switzerland gave more favourable evaluation scores than reviewers affiliated with Swiss institutions. Finally, male reviewers awarded higher evaluation scores than female reviewers and male applicants received more favourable evaluation scores than female applicants. When we controlled for confounding factors, adjusted differences changed little for source of nomination and country of affiliation. In contrast, the gender differences nearly disappeared, which indicates that most of the gender effects observed in univariable analysis is explained by differences in scores between research topics and applicant affiliations. Conclusions: Our study showed that peer review of grant applications at SNSF may be prone to biases stemming from different applicant and reviewer characteristics. Based on this study the SNSF abandoned nomination of reviewers by applicants, and made members of panels aware of the other systematic differences in scores. We encourage other public funding bodies to conduct similar studies.