ObjectivesTo examine whether the gender of applicants and peer reviewers and other factors influence peer review of grant proposals submitted to a national funding agency.SettingSwiss National Science Foundation (SNSF).DesignCross-sectional analysis of peer review reports submitted from 2009 to 2016 using linear mixed effects regression models adjusted for research topic, applicant’s age, nationality, affiliation and calendar period.ParticipantsExternal peer reviewers.Primary outcome measureOverall score on a scale from 1 (worst) to 6 (best).ResultsAnalyses included 38 250 reports on 12 294 grant applications from medicine, architecture, biology, chemistry, economics, engineering, geology, history, linguistics, mathematics, physics, psychology and sociology submitted by 26 829 unique peer reviewers. In univariable analysis, male applicants received more favourable evaluation scores than female applicants (+0.18 points; 95% CI 0.14 to 0.23), and male reviewers awarded higher scores than female reviewers (+0.11; 95% CI 0.08 to 0.15). Applicant-nominated reviewers awarded higher scores than reviewers nominated by the SNSF (+0.53; 95% CI 0.50 to 0.56), and reviewers from outside of Switzerland more favourable scores than reviewers affiliated with Swiss institutions (+0.53; 95% CI 0.49 to 0.56). In multivariable analysis, differences between male and female applicants were attenuated (+0.08; 95% CI 0.04 to 0.13) whereas results changed little for source of nomination and affiliation of reviewers. The gender difference increased after September 2011, when new evaluation forms were introduced (p=0.033 from test of interaction).ConclusionsPeer review of grant applications at SNSF might be prone to biases stemming from different applicant and reviewer characteristics. The SNSF abandoned the nomination of peer reviewers by applicants. The new form introduced in 2011 may inadvertently have given more emphasis to the applicant’s track record. We encourage other funders to conduct similar studies, in order to improve the evidence base for rational and fair research funding.

Background: The Swiss National Science Foundation (SNSF) supports fundamental and use-inspired research in all academic disciplines. As part of the evaluation procedure, grant applications to the SNSF are reviewed by external reviewers. The legitimacy of funding decisions depends on its ability to base funding decisions solely on the scientific merit of grant applications. Aim: We examined whether the following factors influenced the scores given to grant applications submitted to the SNSF: (1) source of nomination of the reviewer, (2) the gender of the applicant and the reviewer, and (3) the country of affiliation of the reviewer. Methods and Data: We gathered data on 38,250 external reviews of 12,294 unique grant applications across all disciplines between 2006 and 2016. Proposals were rated on a scale from 1 (=poor) to 6 (=outstanding) by 26,836 reviewers. We used linear mixed effects regression models adjusted for research topic, applicant’s age, nationality and affiliation. Results: We found that in univariable analysis applicant-nominated reviewers awarded higher evaluation scores than reviewers nominated by the SNSF. Further, reviewers affiliated with research institutions outside of Switzerland gave more favourable evaluation scores than reviewers affiliated with Swiss institutions. Finally, male reviewers awarded higher evaluation scores than female reviewers and male applicants received more favourable evaluation scores than female applicants. When we controlled for confounding factors, adjusted differences changed little for source of nomination and country of affiliation. In contrast, the gender differences nearly disappeared, which indicates that most of the gender effects observed in univariable analysis is explained by differences in scores between research topics and applicant affiliations. Conclusions: Our study showed that peer review of grant applications at SNSF may be prone to biases stemming from different applicant and reviewer characteristics. Based on this study the SNSF abandoned nomination of reviewers by applicants, and made members of panels aware of the other systematic differences in scores. We encourage other public funding bodies to conduct similar studies.

Background The Swiss National Science Foundation (SNSF) supports fundamental and use-inspired research in all disciplines. Peer reviewers assess the proposals submitted to the SNSF. We examined whether the gender of applicants and reviewers and other factors influenced the summary scores awarded. Methods We analysed 38,250 reports on 12,294 grant applications across all disciplines 2006 to 2016. Proposals were rated on a scale from 1 (=worst) to 6 (=best) by 26,836 reviewers. We used linear mixed effects regression models adjusted for research topic, applicant's age, nationality, affiliation and calendar period to examine associations, and interactions between gender of the applicant and other variables. Results In univariable analysis, male applicants received more favourable evaluation scores than female applicants (+0.19 points; 95% CI 0.14-0.23), and male reviewers awarded higher scores than female reviewers (+0.12; 95% CI 0.08-0.15). Applicant-nominated reviewers awarded higher scores than reviewers nominated by the SNSF (+0.53; 95% CI 0.50-0.56), and reviewers affiliated with research institutions outside of Switzerland more favourable scores than reviewers affiliated with Swiss institutions (+0.53; 95% CI 0.49-0.56). In multivariable analysis, differences between male and female applicants were attenuated (to +0.08; 95% CI 0.04-0.13) whereas results changed little for source of nomination and affiliation of reviewers. There was an interaction between gender of applicant and reviewer, and between gender of applicant and calendar period. Male reviewers gave higher scores than female reviewers, with a greater difference for male than for female applicants (P=0.037 from test of interaction). The gender difference increased after September 2011, when new evaluation forms were introduced (P=0.033 from test of interaction). Conclusions Our study showed that peer review of grant applications at SNSF might be prone to biases stemming from different applicant and reviewer characteristics. The SNSF abandoned the nomination of peer reviewers by applicants, and made members of panels aware of the other systematic differences in scores. The new form introduced in 2011 may inadvertently have given more emphasis to the applicant’s track record, and a revision is now under discussion. We encourage other funders to conduct similar studies, in order to improve the evidence base for rational and fair research funding.

Background The Swiss National Science Foundation (SNSF) supports fundamental and use-inspired research in all disciplines. Peer reviewers assess the proposals submitted to the SNSF. We examined whether the gender of applicants and reviewers and other factors influenced the summary scores awarded. Methods We analysed 38,250 reports on 12,294 grant applications across all disciplines 2006 to 2016. Proposals were rated on a scale from 1 (=worst) to 6 (=best) by 26,836 reviewers. We used linear mixed effects regression models adjusted for research topic, applicant's age, nationality, affiliation and calendar period to examine associations, and interactions between gender of the applicant and other variables. Results In univariable analysis, male applicants received more favourable evaluation scores than female applicants (+0.19 points; 95% CI 0.14-0.23), and male reviewers awarded higher scores than female reviewers (+0.12; 95% CI 0.08-0.15). Applicant-nominated reviewers awarded higher scores than reviewers nominated by the SNSF (+0.53; 95% CI 0.50-0.56), and reviewers affiliated with research institutions outside of Switzerland more favourable scores than reviewers affiliated with Swiss institutions (+0.53; 95% CI 0.49-0.56). In multivariable analysis, differences between male and female applicants were attenuated (to +0.08; 95% CI 0.04-0.13) whereas results changed little for source of nomination and affiliation of reviewers. There was an interaction between gender of applicant and reviewer, and between gender of applicant and calendar period. Male reviewers gave higher scores than female reviewers, with a greater difference for male than for female applicants (P=0.037 from test of interaction). The gender difference increased after September 2011, when new evaluation forms were introduced (P=0.033 from test of interaction). Conclusions Our study showed that peer review of grant applications at SNSF might be prone to biases stemming from different applicant and reviewer characteristics. The SNSF abandoned the nomination of peer reviewers by applicants, and made members of panels aware of the other systematic differences in scores. The new form introduced in 2011 may inadvertently have given more emphasis to the applicant’s track record, and a revision is now under discussion. We encourage other funders to conduct similar studies, in order to improve the evidence base for rational and fair research funding.

Background: The Swiss National Science Foundation (SNSF) supports fundamental and use-inspired research in all academic disciplines. As part of the evaluation procedure, grant applications to the SNSF are reviewed by external reviewers. The legitimacy of funding decisions depends on its ability to base funding decisions solely on the scientific merit of grant applications. Aim: We examined whether the following factors influenced the scores given to grant applications submitted to the SNSF: (1) source of nomination of the reviewer, (2) the gender of the applicant and the reviewer, and (3) the country of affiliation of the reviewer. Methods and Data: We gathered data on 38,250 external reviews of 12,294 unique grant applications across all disciplines between 2006 and 2016. Proposals were rated on a scale from 1 (=poor) to 6 (=outstanding) by 26,836 reviewers. We used linear mixed effects regression models adjusted for research topic, applicant’s age, nationality and affiliation. Results: We found that in univariable analysis applicant-nominated reviewers awarded higher evaluation scores than reviewers nominated by the SNSF. Further, reviewers affiliated with research institutions outside of Switzerland gave more favourable evaluation scores than reviewers affiliated with Swiss institutions. Finally, male reviewers awarded higher evaluation scores than female reviewers and male applicants received more favourable evaluation scores than female applicants. When we controlled for confounding factors, adjusted differences changed little for source of nomination and country of affiliation. In contrast, the gender differences nearly disappeared, which indicates that most of the gender effects observed in univariable analysis is explained by differences in scores between research topics and applicant affiliations. Conclusions: Our study showed that peer review of grant applications at SNSF may be prone to biases stemming from different applicant and reviewer characteristics. Based on this study the SNSF abandoned nomination of reviewers by applicants, and made members of panels aware of the other systematic differences in scores. We encourage other public funding bodies to conduct similar studies.