Past performance, peer review and project selection: a case study in the social and behavioral sciences

Abstract

Does past performance influence success in grant applications? In this study we test whether the grant allocation decisions of the Netherlands Research Council for the Economic and Social Sciences correlate with the past performances of the applicants in terms of publications and citations, and with the results of the peer review process organized by the Council. We show that the Council is successful in distinguishing grant applicants with above-average performance from those with below-average performance, but within the former group no correlation could be found between past performance and receiving a grant. When comparing the best performing researchers who were denied funding with the group of researchers who received it, the rejected researchers significantly outperformed the funded ones. Furthermore, the best rejected proposals score on average as high on the outcomes of the peer review process as the accepted proposals. Finally, we found that the Council under study successfully corrected for gender effects during the selection process. We explain why these findings may be more general than for this case only. However, if research councils are not able to select the 'best' researchers, perhaps they should reconsider their mission. In a final section with policy implications, we discuss the role of research councils at the level of the science system in terms of variation, innovation, and quality control.

Organizational factors influencing scholarly performance: a multivariate study of biomedical research groups

“…To overcome the constraints of institutional funds, research groups increasingly try to acquire funding from a variety of external (and increasingly also international) sources (Geuna 2001). Yet, funding sources have different expectations from the work they fund and elicit different strategies from applicants: Competitive research council funding focuses on stimulating academic careers (Bornmann et al 2010;Hornbostel et al 2009;Van den Besselaar and Leydesdorff 2009), and industry funding aims at useful (applied) research results (Groot and García-Valderrama 2006;Gulbrandsen and Smeby 2005;Carayol 2003;Louis et al 2007). However, quite some studies did not find a relation between funding levels and performance (Carayol and Matt 2006;Cherchye and Abeele 2005;Groot and García-Valderrama 2006).…”

External Tests of Peer Review Validity Via Impact Measures

“…A couple of studies examined the ex-ante productivity of applicants relative to review scores, and found significant correlations, as well as significant biases (Wenneras and Wold, 1997;Sandstrom and Hallsten, 2008). Also, some studies show when you compare the best of unfunded applicants with funded ex ante, they are comparable (van den Besselaar and Leydesdorff, 2009;Bornmann et al, 2010;Neufeld et al, 2013), suggesting significant type II error. Some of these studies have been summarized well by Boyack et al (2018) as well as Van den Besselaar and Sandstrom (2015).…”

“…This is a powerful strategy as you can directly compare funded and unfunded applicants, and do not have to consider the effect of funding as a confounding factor on performance. Most studies show that overall funded applicants outperform unfunded (Bornmann and Daniel, 2006;van den Besselaar and Leydesdorff, 2009;Bornmann et al, 2010;van Leeuwen and Moed, 2012;Cabezas-Clavijo et al, 2013) and a few studies do not (Hornbostel et al, 2009;Neufeld et al, 2013;Saygitov, 2014), although typically the differences are small and dependent on the general quality level of applicants (if all applicants are very productive, smaller differences will be observed). A couple of studies examined the ex-ante productivity of applicants relative to review scores, and found significant correlations, as well as significant biases (Wenneras and Wold, 1997;Sandstrom and Hallsten, 2008).…”

What is the Required Level of Data Cleaning? A Research Evaluation Case

“…An earlier study [7] on the social science division of aresearch council found that applicants out performnonapplicants and that successful applicants perform better than nonselected applicants (NS) and that the best performing nonsuccessful applicants (BPNS) tend to perform on average at least as good as the successful applicants (S), actually leading to high percentages of false positives and false negatives. Best performing can be defined in different ways.…”

“…In contrast to the 2009 study, [7] we here not only compare averages but also, as the data are rather skewed, compare the medians. Table 2 shows the results for the sample of 905 applications, for the original 2007 data, and the new manually collected and corrected data.…”