Predicting Technology Success: Identifying Key Predictors and Assessing Expert Evaluation for Advanced Technologies

Galbraith, Craig S.; Ehrlich, Sanford B.; DeNoble, Alex F.

doi:10.1007/s10961-006-0022-8

Cited by 43 publications

(49 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Galbraith et al (2006) also compared peer review scores of individual funded projects to their ultimate success utilizing two metrics: (1) cooperative research and development agreements (CRADA) or licenses that were signed, SBIR or equity funding that was obtained or a product that was launched; and (2) the assessment of a senior project manager (not an author) of each technology as successful (evaluated one and a half to 3 years after the initial peer review evaluation). Using 69 early to mid-stage homeland defense technologies funded by the US DoD Center for Commercialization of Advanced Technologies (CCAT), the authors found that reviewer scores were weakly predictive of commercial success of funded projects.…”

Section: Patents/technology Developmentmentioning

confidence: 99%

External Tests of Peer Review Validity Via Impact Measures

Gallo

Glisson

2018

Front. Res. Metr. Anal.

View full text Add to dashboard Cite

Peer review is used commonly across science as a tool to evaluate the merit and potential impact of research projects and make funding recommendations. However, potential impact is likely to be difficult to assess ex-ante; some attempts have been made to assess the predictive accuracy of these review decisions using impact measures of the results of the completed projects. Although many outputs, and thus potential measures of impact, exist for research projects, the overwhelming majority of evaluation of research output is focused on bibliometrics. We review the multiple types of potential impact measures with an interest in their application to validate review decisions. A review of the current literature on validating peer review decisions with research output impact measures is presented here; only 48 studies were identified, about half of which were US based and sample size per study varied greatly. 69% of the studies employed bibliometrics as a research output. While 52% of the studies employed alternative measures (like patents and technology licensing, post-project peer review, international collaboration, future funding success, securing tenure track positions, and career satisfaction), only 25% of all projects used more than one measure of research output. Overall, 91% of studies with unfunded controls and 71% of studies without such controls provided evidence for at least some level of predictive validity of review decisions. However, several studies reported observing sizable type I and II errors as well. Moreover, many of the observed effects were small and several studies suggest a coarse power to discriminate poor proposals from better ones, but not amongst the top tier proposals or applicants (although discriminatory ability depended on the impact metric). This is of particular concern in an era of low funding success, where many top tier proposals are unfunded. More research is needed, particularly in integrating multiple types of impact indicators in these validity tests, as well as considering the context of the research outputs relative to goals of the research program and concerns for reproducibility, translatability and publication bias. In parallel, more research is needed focusing on the internal validity of review decision making procedures and reviewer bias.Keywords: research funding, peer review, impact metrics, validity, grant GRANT PEER REVIEW AND IMPACT ASSESSMENTMost would generally agree the purpose of biomedical research is to advance knowledge for societal benefit, with the hope of favorably impacting disease outcomes and improving global health. Indeed, the National Institutes of Health (NIH ), the world's largest funder of biomedical research, characterizes their mission as to "seek fundamental knowledge about the nature and Gallo and GlissonReview Validity via Impact Measures behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability" (NIH, 2017). To help select which research projects to fund to achieve...

show abstract

Section: Patents/technology Developmentmentioning

confidence: 99%

External Tests of Peer Review Validity Via Impact Measures

Gallo

Glisson

2018

Front. Res. Metr. Anal.

View full text Add to dashboard Cite

show abstract

“…Thus, market potential reflects the likely economic or social value that can be generated by a new product or service (de Vries 2012). With respect to innovation readiness, successful launch of innovative products or services begins with the identification of technologies that are ready for commercialisation (Galbraith et al 2006;Heslop et al 2001). Frequently, innovation potential of innovations is assessed as low for novel technologies, at early stages of development.…”

Section: Innovation Potential Assessment Frameworkmentioning

confidence: 99%

Organizational diversity and innovation potential of EU-funded research projects

Nepelski

Piroli

2017

J Technol Transf

View full text Add to dashboard Cite

Impact evaluations of collaborative research projects usually focus on private benefits of participants, e.g. their turnover or employment growth. We study the innovative performance of collaborative research projects and how it depends on the organizational diversity of participating organizations. Our population includes participants to EC-funded collaborative research projects that are considered as key organisations behind delivering innovations. The focus on innovative rather than, for example, financial outcomes allows us to assess the transformative effect of publically-funded collaborative research. We show that the innovative potential of research output of homogenous partnerships, e.g. between two SMEs or two large companies, is likely to be higher, as compared to heterogeneous partnerships, e.g. an SME and a large company. The impact of universities on the potential of innovations is unclear. The total number of key organizations in delivering an innovation has negative impact on its potential. Neither project funding nor duration affects the potential of innovation. Our results implicitly show that, depending on the type of organization and consortium design, there are different incentives to contribute to innovative efforts and opportunities to appropriate their benefits.

show abstract

“…También declaran que la red de contactos del inventor es crucial para la mercadotecnia de la invención. Más aún Galbraith et al (Galbraith, Ehrlich, & DeNoble, 2006) hablan acerca de la importancia de las alianzas estratégicas con compañías para el éxito de la tecnología.…”

Section: Variables Claveunclassified

Selección del portafolio de proyectos tecnológicos en la etapa temprana de la innovación. Desarrollo de una herramienta de evaluación

Güemes‐Castorena

Castillo

2014

View full text Add to dashboard Cite

and implement an easy and practical evaluation tool. The result of this research was that the identified variables were grouped in five categories, as well as the questions to be asked in order to evaluate the technology. Also, the relative weight of each category is presented based on paired comparisons. The tool has been implemented and has helped to identify commercial success characteristics of the technological projects.

show abstract

Predicting Technology Success: Identifying Key Predictors and Assessing Expert Evaluation for Advanced Technologies

Cited by 43 publications

References 33 publications

External Tests of Peer Review Validity Via Impact Measures

External Tests of Peer Review Validity Via Impact Measures

Organizational diversity and innovation potential of EU-funded research projects

Selección del portafolio de proyectos tecnológicos en la etapa temprana de la innovación. Desarrollo de una herramienta de evaluación

Contact Info

Product

Resources

About