Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance, Novelty, and Resource Allocation in Science

Abstract

Selecting among alternative projects is a core management task in all innovating organizations. In this paper, we focus on the evaluation of frontier scientific research projects. We argue that the “intellectual distance” between the knowledge embodied in research proposals and an evaluator’s own expertise systematically relates to the evaluations given. To estimate relationships, we designed and executed a grant proposal process at a leading research university in which we randomized the assignment of evaluators and proposals to generate 2,130 evaluator–proposal pairs. We find that evaluators systematically give lower scores to research proposals that are closer to their own areas of expertise and to those that are highly novel. The patterns are consistent with biases associated with boundedly rational evaluation of new ideas. The patterns are inconsistent with intellectual distance simply contributing “noise” or being associated with private interests of evaluators. We discuss implications for policy, managerial intervention, and allocation of resources in the ongoing accumulation of scientific knowledge.

Risk evaluation in peer review of grant applications

“…While it is possible that reviewers may have taken riskiness into account in their scores, they do not seem to be clearly expressing these concerns in their review comments. It is more likely that the bias is real and that reviewers are simply unaware of their risk aversion, as previous studies have documented the penalization of highly innovative work in a well-controlled peer review system (Boudreau et al 2016).…”

“…Some have suggested that highly innovative research may be associated with more reviewer uncertainty about their judgments of the methodology, which may lead to lower scores (Luukkonen 2012;Boudreau et al 2016). While reviewers may consider innovation in their decisions, they likely give this less weight than methodological weaknesses, which is reflected in the content of the reviewer's critique.…”

“…Some have suggested that reviewers may judge innovative research as being less rigorous due to its novelty; therefore, reviewers are less inclined to tolerate risks and evaluate innovative proposals favorably, which is supported by recent criteria score analyses (Luukkonen 2012; Eblen et al 2016). While reviewers likely use their subject matter expertize to identify innovative projects, some explorations have suggested that even when expertize is controlled, highly novel proposals are routinely penalized in score (Boudreau et al 2016). Thus, if one of the goals of funding agencies is to "foster fundamental creative discoveries, innovative research strategies, and their applications" (NIH 2013), this may not be achieved, potentially due to reviewer risk aversion, although it is unclear if reviewers perceive themselves as avoiding risk.…”

Made in Academia: The Effect of Institutional Origin on Inventors’ Attention to Science

“…Differing evaluation criteria are possible, which can lead to controversies (Engelhardt and Caplan 1987;Boudreau et al 2016). In the past few years, practitioners, the media, and many scientists have loudly complained that there is no mechanism by which inventors can guarantee the usability of academic findings (The Economist 2013;Freedman, Cockburn, and Simcoe 2015).…”

A manifesto for fair and equitable research funding in ecology

“…Known negative biases include those driven by gender (Pohlhaus 2011;Tricco et al 2017), race (Ginther et al 2011), institution size (Murray et al 2016), geographic location (Wahls 2016), and interdisciplinary study (Bromham et al 2016). Additionally, Boudreau et al (2016) found a tendency for reviewers with a shorter intellectual distance from an application to provide harsher evaluations even when they were highly innovative. Other factors can also influence decision making, such as the bias towards the perceived value of high-income-country research (Harris et al 2017).…”