External Tests of Peer Review Validity Via Impact Measures

Abstract

Peer review is used commonly across science as a tool to evaluate the merit and potential impact of research projects and make funding recommendations. However, potential impact is likely to be difficult to assess ex-ante; some attempts have been made to assess the predictive accuracy of these review decisions using impact measures of the results of the completed projects. Although many outputs, and thus potential measures of impact, exist for research projects, the overwhelming majority of evaluation of research output is focused on bibliometrics. We review the multiple types of potential impact measures with an interest in their application to validate review decisions. A review of the current literature on validating peer review decisions with research output impact measures is presented here; only 48 studies were identified, about half of which were US based and sample size per study varied greatly. 69% of the studies employed bibliometrics as a research output. While 52% of the studies employed alternative measures (like patents and technology licensing, post-project peer review, international collaboration, future funding success, securing tenure track positions, and career satisfaction), only 25% of all projects used more than one measure of research output. Overall, 91% of studies with unfunded controls and 71% of studies without such controls provided evidence for at least some level of predictive validity of review decisions. However, several studies reported observing sizable type I and II errors as well. Moreover, many of the observed effects were small and several studies suggest a coarse power to discriminate poor proposals from better ones, but not amongst the top tier proposals or applicants (although discriminatory ability depended on the impact metric). This is of particular concern in an era of low funding success, where many top tier proposals are unfunded. More research is needed, particularly in integrating multiple types of impact indicators in these validity tests, as well as considering the context of the research outputs relative to goals of the research program and concerns for reproducibility, translatability and publication bias. In parallel, more research is needed focusing on the internal validity of review decision making procedures and reviewer bias.Keywords: research funding, peer review, impact metrics, validity, grant GRANT PEER REVIEW AND IMPACT ASSESSMENTMost would generally agree the purpose of biomedical research is to advance knowledge for societal benefit, with the hope of favorably impacting disease outcomes and improving global health. Indeed, the National Institutes of Health (NIH ), the world's largest funder of biomedical research, characterizes their mission as to "seek fundamental knowledge about the nature and Gallo and GlissonReview Validity via Impact Measures behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability" (NIH, 2017). To help select which research projects to fund to achieve...

Questionable research practices in competitive grant funding: A survey

“…When researchers express doubts about these aspects, it indicates a potentially larger problem. It is also noteworthy that these findings are in line with existing evidence that there is little or no relation between the ranking of projects by peer review and long-term scientific success [47][48][49][50], that review scores differ substantially between reviewers [51,52], and that acquiring funding highly depends on which particular reviewers do the review [53,54].…”

Peer review: Risk and risk tolerance

“…Mann-Whitney or Kruskal-Wallis tests) were conducted to confirm differences between risk scenarios using StatPlus TM software. To examine how well participants could detect risk, receiver operating characteristic (ROC) curves were calculated for the overall score using a series of thresholds over the full scoring range (1)(2)(3)(4)(5)(6)(7)(8)(9). We defined a true positive as a participant identifying correctly that a scenario is risky by scoring worse that the threshold.…”

“…The pilot subjects were also interviewed and based on their feedback, the OISs were modified slightly. The pilot study and the study reported here were reviewed by the Washington State University Office of Research Assurances (Assurance# FWA00002946) and granted exemptions from IRB review consistent with 45 CFR 46.104(d) (2).…”

“…Research funding agencies, including the US National Institutes for Health (NIH), rely on peer review to help identify the most scientifically meritorious and impactful research proposals [1]. Of concern is evidence that reviewers can discriminate good from bad proposals more reliably then good from great proposals [2][3][4]. Reviewers are often expected to evaluate beyond their discriminatory power, resulting in relying on biases and preferences [5,6].…”