A Multilevel Cross-Classified Modelling Approach to Peer Review of Grant Proposals: The Effects of Assessor and Researcher Attributes on Assessor Ratings

Abstract

The peer review of grant proposals is very important to academics from all disciplines. Although there is limited research on the reliability of assessments for grant proposals, previously reported single-rater reliabilities have been disappointingly low (between 0.17 and 0.37). We found that the single-rater reliability of the overall assessor rating for Australian Research Council grants was 0.21 for social science and humanities (2870 ratings, 1928 assessors and 687 proposals) and 0.19 for science (7153 ratings, 4295 assessors and 1644 proposals). We used a multilevel, cross-classification approach (level 1, assessor and proposal cross-classification; level 2, field of study), taking into account that 34% of the assessors evaluated more than one proposal. Researcher-nominated assessors (those chosen by the authors of the research proposal) gave higher ratings than panel-nominated assessors chosen by the Australian Research Council, and proposals from more prestigious universities received higher ratings. In the social sciences and humanities, the status of Australian universities had significantly more effect on Australian assessors than on overseas assessors. In science, ratings were higher when assessors rated fewer proposals and apparently had a more limited frame of reference for making such ratings and when researchers were professors rather than non-professors. Particularly, the methodology of this large scale study is applicable to other forms of peer review (publications, job interviews, awarding of prizes and election to prestigious societies) where peer review is employed as a selection process. Copyright 2003 Royal Statistical Society.

Does the Gender Composition of Scientific Committees Matter?

“…Sometimes researchers seem to benefit from the presence of evaluators who share the same gender (Casadevall andHandelsman 2013, De Paola andScoppa 2014), sometimes applicants seem to obtain relatively better evaluations from opposite-sex evaluators (Broder 1993;Ellemers, Heuvel, de Gilder, Maass and Bonvini 2004), and in some other cases gender does not seem to play any (statistically) significant role (Abrevaya and Hamermesh 2012;Jayasinghe, Marsh and Bond 2003;Moss-Racusin, Dovidio, Brescoll, 5 In 1995 gender quotas were introduced in Finland through the amendment of the Finnish Act on Equality between Women and Men. In 1999, the European Commission stated the aim to achieve at least a 40% representation of women in Marie Curie scholarships, advisory groups, assessment panels and monitoring panels (COM(1999) 76 Final).…”

Do Gender Quotas Pass the Test? Evidence from Academic Evaluations in Italy

“…Female faculty members are most inclined to hold stereotypical views. Jayasinghe, Marsh and Bond (2003) analyze the ratings obtained by 687 grant proposals at the Australian Research Council. They do not find any evidence suggesting that assessors favor same-sex or opposite sex researchers.…”

“…3 However, the empirical evidence on the impact of committees' gender composition is scarce, typically based on small samples, and it is rather inconclusive. Sometimes, applicants seem to benefit from the presence of same-sex evaluators (Li 2012, De Paola and, but most often gender does not seem to play any role (Moss-Racusin, Dovidio, Brescoll, Graham and Handelsman 2012;Steinpreis, Anders and Ritzke 1999;Abrevaya and Hamermesh 2012;Jayasinghe, Marsh and Bond 2003) or applicants obtain relatively better evaluations from opposite-sex evaluators (Broder 1993;Ellemers, Heuvel, de Gilder, Maass and Bonvini 2004). From a policy perspective, the lack of more extensive and conclusive evidence is disappointing.…”

Reliability of reviewers' ratings when using public peer review: a case study

“…21 However, the ICC also finds use in multilevel regression analysis. 23 Unlike the random-effects ANOVA model, here the assumption can be relaxed that the dependent variable has an interval level and is normally distributed. 21 In the case of a categorical variable with ordered categories (here: the reviewers' ratings) a random-intercept proportional odds model can be used.…”