Potential bias in peer review of grant applications at the Swiss National Science Foundation

Abstract

Background: The Swiss National Science Foundation (SNSF) supports fundamental and use-inspired research in all academic disciplines. As part of the evaluation procedure, grant applications to the SNSF are reviewed by external reviewers. The legitimacy of funding decisions depends on its ability to base funding decisions solely on the scientific merit of grant applications. Aim: We examined whether the following factors influenced the scores given to grant applications submitted to the SNSF: (1) source of nomination of the reviewer, (2) the gender of the applicant and the reviewer, and (3) the country of affiliation of the reviewer. Methods and Data: We gathered data on 38,250 external reviews of 12,294 unique grant applications across all disciplines between 2006 and 2016. Proposals were rated on a scale from 1 (=poor) to 6 (=outstanding) by 26,836 reviewers. We used linear mixed effects regression models adjusted for research topic, applicant’s age, nationality and affiliation. Results: We found that in univariable analysis applicant-nominated reviewers awarded higher evaluation scores than reviewers nominated by the SNSF. Further, reviewers affiliated with research institutions outside of Switzerland gave more favourable evaluation scores than reviewers affiliated with Swiss institutions. Finally, male reviewers awarded higher evaluation scores than female reviewers and male applicants received more favourable evaluation scores than female applicants. When we controlled for confounding factors, adjusted differences changed little for source of nomination and country of affiliation. In contrast, the gender differences nearly disappeared, which indicates that most of the gender effects observed in univariable analysis is explained by differences in scores between research topics and applicant affiliations. Conclusions: Our study showed that peer review of grant applications at SNSF may be prone to biases stemming from different applicant and reviewer characteristics. Based on this study the SNSF abandoned nomination of reviewers by applicants, and made members of panels aware of the other systematic differences in scores. We encourage other public funding bodies to conduct similar studies.

The leaky pipeline in research grant peer review and funding decisions: challenges and future directions

“…Some studies have also shown that male reviewers were prone to give, overall, higher evaluations than female reviewers (e.g. Broder, 1993;Jayasinghe et al, 2003;Severin et al, 2019), although this effect was found to be rather modest (Severin et al, 2019). Additionally, the common assumption that external reviewers may hold preferences for applicants with the same or a specific gender (Jayasinghe et al, 2003) has been examined in grant schemes for the Australian Research Council (Jayasinghe et al, 2001(Jayasinghe et al, , 2003Marsh et al, 2008Marsh et al, , 2011, the Austrian Science Fund (Mutz et al, 2012), the German Boehringer Ingelheim Fonds , and the Swiss National Science Foundation (Severin et al, 2019), and none have substantiated the effect.…”

“…In addition to the insufficient number of eligible senior researchers, women, compared to men, are generally less inclined to apply for grant funding or renewal (Beck & Halloin, 2017;Blake & La Valle, 2000;Boyle et al, 2015;Friesen, 1998;Grant et al, 1997;Jayasinghe et al, 2003;Marsh et al, 2008;Pohlhaus et al, 2011;Severin et al, 2019;Ward & Donnelly, 1998). On a practical level, institutional support may not be available for eligible women to relieve them of administrative or teaching duties to prepare for the grant writing process or to support them in receiving the necessary training needed for preparing applications (Easterly & Pemberton, 2008).…”

“…dermatology research: Cheng et al, 2016 ; global infectious disease research: Head et al, 2013 ; cognitive science: Titone et al, 2018 ; cancer research: Zhou et al, 2018 ) and within various national contexts (e.g. UK: Blake & La Valle, 2000 ; Netherlands: Brouns, 2000 ; van der Lee & Ellemers, 2015 ; USA: Eloy et al, 2013 ; Jagsi et al, 2009 ; Australia: Over, 1996 ; Sigelman & Scioli, 1987 ; Canada: Burns et al, 2019 ; Tamblyn, et al, 2018 ; Switzerland: Severin et al, 2019 ). Female applicants have been shown to be disfavoured compared to their male counterparts, resulting in, proportionally, fewer of their studies being funded (Gannon et al, 2001 ; Head et al, 2013 ; Jagsi et al, 2009 ; Steinþórsdóttir et al, 2019 ; Zhou et al, 2018 ), fewer requests being awarded (Waisbren et al, 2008 ), and lower funding amounts being allotted (Bedi et al, 2012 ; Eloy et al, 2013 ; Steinþórsdóttir et al, 2019 ; Zhou et al, 2018 ).…”

“…Broder, 1993 ; Jayasinghe et al, 2003 ; Severin et al, 2019 ), although this effect was found to be rather modest (Severin et al, 2019 ). Additionally, the common assumption that external reviewers may hold preferences for applicants with the same or a specific gender (Jayasinghe et al, 2003 ) has been examined in grant schemes for the Australian Research Council (Jayasinghe et al, 2001 , 2003 ; Marsh et al, 2008 , 2011 ), the Austrian Science Fund (Mutz et al, 2012 ), the German Boehringer Ingelheim Fonds (Bornmann & Daniel, 2007 ), and the Swiss National Science Foundation (Severin et al, 2019 ), and none have substantiated the effect. Thus, no interaction effects between applicant gender and reviewer gender have been found thus far (Bornmann & Daniel, 2007 ; Jayasinghe et al, 2001 ; Marsh et al, 2011 ; Mutz et al, 2012 ), with these results being considered as generalisable across disciplines (Marsh et al, 2008 , 2011 ).…”

Are peer reviews of grant proposals reliable? An analysis of Economic and Social Research Council (ESRC) funding applications

“…Criticisms particularly focus on a lack of consistency between reviewers in their evaluations of research quality (e.g. Bornman, Mutz & Daniel, 2010;Pier et al, 2018;Smith, 2006), and on the potential scope for bias in reviews (Marsh, Jayasinghe and Bond 2008;Severin, Martins, Delavy, Jorstad, & Egger, 2019). This paper uses heretofore unpublished data from the largest social-science funder in the UK (the Economic and Social Research Council; ESRC) to investigate issues of consistency and bias in peer-reviews of funding proposals.…”