The applied value of public investments in biomedical research

Li, Danielle; Azoulay, Pierre; Sampat, Bhaven N.

doi:10.1126/science.aal0010

Cited by 146 publications

(110 citation statements)

References 44 publications

Supporting

Mentioning

102

Contrasting

Unclassified

Order By: Relevance

“…Patents have been used as indicators of research impact, although some studies find that only about 10% of NIH grants over the last 3 decades directly yielded a patent as a product and only about 30% have work which is cited in a patent (Li et al, 2017). Other studies have shown that, to bring 5 patented therapeutics through testing and to the market required more than 100,000 papers, and nearly 20,000 NIH grants (Keserci et al, 2017; other funding sources not considered).…”

Section: Patents/technology Developmentmentioning

confidence: 99%

External Tests of Peer Review Validity Via Impact Measures

Gallo

Glisson

2018

Front. Res. Metr. Anal.

View full text Add to dashboard Cite

Peer review is used commonly across science as a tool to evaluate the merit and potential impact of research projects and make funding recommendations. However, potential impact is likely to be difficult to assess ex-ante; some attempts have been made to assess the predictive accuracy of these review decisions using impact measures of the results of the completed projects. Although many outputs, and thus potential measures of impact, exist for research projects, the overwhelming majority of evaluation of research output is focused on bibliometrics. We review the multiple types of potential impact measures with an interest in their application to validate review decisions. A review of the current literature on validating peer review decisions with research output impact measures is presented here; only 48 studies were identified, about half of which were US based and sample size per study varied greatly. 69% of the studies employed bibliometrics as a research output. While 52% of the studies employed alternative measures (like patents and technology licensing, post-project peer review, international collaboration, future funding success, securing tenure track positions, and career satisfaction), only 25% of all projects used more than one measure of research output. Overall, 91% of studies with unfunded controls and 71% of studies without such controls provided evidence for at least some level of predictive validity of review decisions. However, several studies reported observing sizable type I and II errors as well. Moreover, many of the observed effects were small and several studies suggest a coarse power to discriminate poor proposals from better ones, but not amongst the top tier proposals or applicants (although discriminatory ability depended on the impact metric). This is of particular concern in an era of low funding success, where many top tier proposals are unfunded. More research is needed, particularly in integrating multiple types of impact indicators in these validity tests, as well as considering the context of the research outputs relative to goals of the research program and concerns for reproducibility, translatability and publication bias. In parallel, more research is needed focusing on the internal validity of review decision making procedures and reviewer bias.Keywords: research funding, peer review, impact metrics, validity, grant GRANT PEER REVIEW AND IMPACT ASSESSMENTMost would generally agree the purpose of biomedical research is to advance knowledge for societal benefit, with the hope of favorably impacting disease outcomes and improving global health. Indeed, the National Institutes of Health (NIH ), the world's largest funder of biomedical research, characterizes their mission as to "seek fundamental knowledge about the nature and Gallo and GlissonReview Validity via Impact Measures behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability" (NIH, 2017). To help select which research projects to fund to achieve...

show abstract

Section: Patents/technology Developmentmentioning

confidence: 99%

External Tests of Peer Review Validity Via Impact Measures

Gallo

Glisson

2018

Front. Res. Metr. Anal.

View full text Add to dashboard Cite

show abstract

“…Patents are often viewed by economists as measures of productivity and innovation (Li et al, 2017) and are also taken into consideration by the NIH when evaluating grant applications. Following legislative changes and landmark court cases regarding patentability of biotechnology in the early 1980s (Katz, 2015;Heller and Eisenberg, 1998), the number of patents stemming from NIH-funded research has increased substantially ( Figure 7A).…”

Section: Minority Of Highly Funded Institutes Own Majority Of Nih-funmentioning

confidence: 99%

“…Since this period, universities have dramatically expanded administrative bodies that oversee faculty research (Ginsberg, 2011), such as technology transfer offices that aim to patent potentially lucrative academic research. Patentability became another metric for evaluating scientific research and success in the patent office is often taken as an indicator of whether scientific research is yielding practical results for society (Li et al, 2017).…”

Section: Introductionmentioning

confidence: 99%

On the Biomedical Elite: Inequality and Stasis in Scientific Knowledge Production

Katz

Matter

2017

SSRN Journal

View full text Add to dashboard Cite

Researchers and research institutes are increasingly being evaluated using metrics (from bibliometrics to patent counts), which are core instruments of a longstanding effort to quantify scientific productivity and worth. Here, we examine the relationship between commonly used metrics and funding levels for investigators funded by the National Institutes of Health, the largest public funder of biomedical research in the United States, in the years 1985-2015. We find that funding inequality has been rising since 1985, with a small segment of investigators and institutes getting an increasing proportion of funds, and that investigators who start in the top funding ranks tend to stay there (which results in stasis, or lack of mobility). Furthermore, funding levels are a strong quantitative predictor of the interrelated set of metrics frequently used by economists and policy makers to evaluate scientific research. Our results suggest that the widespread system of metrics favors a minority of elite, highly funded researchers and institutes. Current attempts to "optimize" science are inextricably linked to the concentration of funds in the biomedical research system and are likely to further reduce diversity in the research community.

show abstract

“…"But we will pay special attention. " ■ 1 . But more than three times that number -30.8% -produced a scientific article that was later cited in a commercial patent for a drug, device or other medical technology.…”

Section: Genetic Linksmentioning

confidence: 99%

“…Last November, they published their results: of 131 people who received anti-PD-1 therapies, 9% developed "hyperprogressive" disease, with accelerated tumour growth 1 . The phenomenon seemed to be more common in people over the age of 65.…”

mentioning

confidence: 99%

NIH research grants yield economic windfall

Dolgin¹

2017

Nature

View full text Add to dashboard Cite

The applied value of public investments in biomedical research

Cited by 146 publications

References 44 publications

External Tests of Peer Review Validity Via Impact Measures

External Tests of Peer Review Validity Via Impact Measures

On the Biomedical Elite: Inequality and Stasis in Scientific Knowledge Production

NIH research grants yield economic windfall

Contact Info

Product

Resources

About