Association of percentile ranking with citation impact and productivity in a large cohort of de novo NIMH-funded R01 grants

Doyle, Jamie Mihoko; Quinn, Kimberly A.; Bodenstein, Y A; Wu, Colin O.; Danthi, Narasimhan; Lauer, Michael S.

doi:10.1038/mp.2015.71

Cited by 36 publications

(32 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A very detailed study shows that percentile scores cannot predict the productivity of almost 7,000 National Heart, Lung, and Blood Institute grants (35). In contrast to work Li and Agha (36, 37), Lauer et al (38) showed that study section scores are not discriminating predictors for proposals (39).…”

mentioning

confidence: 96%

Future of fundamental discovery in US biomedical research

Levitt

2017

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Young researchers are crucially important for basic science as they make unexpected, fundamental discoveries. Since 1982, we find a steady drop in the number of grant-eligible basic-science faculty [principal investigators (PIs)] younger than 46. This fall occurred over a 32-y period when inflation-corrected congressional funds for NIH almost tripled. During this time, the PI success ratio (fraction of basic-science PIs who are R01 grantees) dropped for younger PIs (below 46) and increased for older PIs (above 55). This age-related bias seems to have caused the steady drop in the number of young basic-science PIs and could reduce future US discoveries in fundamental biomedical science. The NIH recognized this bias in its 2008 early-stage investigator (ESI) policy to fund young PIs at higher rates. We show this policy is working and recommend that it be enhanced by using better data. Together with the National Institute of General Medical Sciences (NIGMS) Maximizing Investigators’ Research Award (MIRA) program to reward senior PIs with research time in exchange for less funding, this may reverse a decades-long trend of more money going to older PIs. To prepare young scientists for increased demand, additional resources should be devoted to transitional postdoctoral fellowships already offered by NIH.

show abstract

mentioning

confidence: 96%

Future of fundamental discovery in US biomedical research

Levitt

2017

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…Thus, it is unclear looking at an individual's career productivity alone may be an appropriate measure of success to validate review decisions; analysis of ex post productivity of individual projects is also required. (Berg, 2011;Fortin and Currie, 2013;Gallo et al, 2014) has predicted diminishing returns with larger investments; these studies found no correlation between review scores and output Doyle et al, 2015). However, a large NIH study of unnormalized bibliometric data found a moderate correlation (Li and Agha, 2015).…”

mentioning

confidence: 99%

External Tests of Peer Review Validity Via Impact Measures

Gallo

Glisson

2018

Front. Res. Metr. Anal.

View full text Add to dashboard Cite

Peer review is used commonly across science as a tool to evaluate the merit and potential impact of research projects and make funding recommendations. However, potential impact is likely to be difficult to assess ex-ante; some attempts have been made to assess the predictive accuracy of these review decisions using impact measures of the results of the completed projects. Although many outputs, and thus potential measures of impact, exist for research projects, the overwhelming majority of evaluation of research output is focused on bibliometrics. We review the multiple types of potential impact measures with an interest in their application to validate review decisions. A review of the current literature on validating peer review decisions with research output impact measures is presented here; only 48 studies were identified, about half of which were US based and sample size per study varied greatly. 69% of the studies employed bibliometrics as a research output. While 52% of the studies employed alternative measures (like patents and technology licensing, post-project peer review, international collaboration, future funding success, securing tenure track positions, and career satisfaction), only 25% of all projects used more than one measure of research output. Overall, 91% of studies with unfunded controls and 71% of studies without such controls provided evidence for at least some level of predictive validity of review decisions. However, several studies reported observing sizable type I and II errors as well. Moreover, many of the observed effects were small and several studies suggest a coarse power to discriminate poor proposals from better ones, but not amongst the top tier proposals or applicants (although discriminatory ability depended on the impact metric). This is of particular concern in an era of low funding success, where many top tier proposals are unfunded. More research is needed, particularly in integrating multiple types of impact indicators in these validity tests, as well as considering the context of the research outputs relative to goals of the research program and concerns for reproducibility, translatability and publication bias. In parallel, more research is needed focusing on the internal validity of review decision making procedures and reviewer bias.Keywords: research funding, peer review, impact metrics, validity, grant GRANT PEER REVIEW AND IMPACT ASSESSMENTMost would generally agree the purpose of biomedical research is to advance knowledge for societal benefit, with the hope of favorably impacting disease outcomes and improving global health. Indeed, the National Institutes of Health (NIH ), the world's largest funder of biomedical research, characterizes their mission as to "seek fundamental knowledge about the nature and Gallo and GlissonReview Validity via Impact Measures behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability" (NIH, 2017). To help select which research projects to fund to achieve...

show abstract

“…Lindner and Nakamura 21 found that the bibliometric indexes of publications from funded projects are not an appropriate measure of the scientific impact of an application. Similarly, Kaltman et al 22 and Doyle et al 23 found no association between grant percentile ranking and an applicant's bibliometric indexes. Derrick et al 24 evaluated whether citation metrics correlated with peer assessment of a researcher's influence on his/her field.…”

Section: The Grant Itselfmentioning

confidence: 94%

Peer Review Practices for Evaluating Biomedical Research Grants: A Scientific Statement From the American Heart Association

Liaw¹,

Freedman²,

Becker³

et al. 2017

Circulation Research

View full text Add to dashboard Cite

The biomedical research enterprise depends on the fair and objective peer review of research grants, leading to the distribution of resources through efficient and robust competitive methods. In the United States, federal funding agencies and foundations collectively distribute billions of dollars annually to support biomedical research. For the American Heart Association, a Peer Review Subcommittee is charged with establishing the highest standards for peer review. This scientific statement reviews the current literature on peer review practices, describes the current American Heart Association peer review process and those of other agencies, analyzes the strengths and weaknesses of American Heart Association peer review practices, and recommends best practices for the future

show abstract

Association of percentile ranking with citation impact and productivity in a large cohort of de novo NIMH-funded R01 grants

Cited by 36 publications

References 13 publications

Future of fundamental discovery in US biomedical research

Future of fundamental discovery in US biomedical research

External Tests of Peer Review Validity Via Impact Measures

Peer Review Practices for Evaluating Biomedical Research Grants: A Scientific Statement From the American Heart Association

Contact Info

Product

Resources

About