Experience with NIH Peer Review: Researchers' Cynicism and Desire for Change

Abstract

In the United States, peer review is central to the process by which many government agencies select research proposals for funding. Although several different agency versions of peer review are practiced,2 they share one characteristic: Scientists judge both the potential value of proposed research projects and the ability of proposers to perform the studies. The premise is self-fulfilling. If scientists identified as peer specialists are best qualified to judge the scientific merit of proposals, then peer review becomes a tool of science policy; it both affects how policy is implemented and how decisions on public funding of research are justified.Many scientists strongly support peer review, at least in the abstract.3 Critics of peer review, however, often suggest that the abstract conception does not hold in practice. There are difficulties, they argue, in defining who are &dquo;peers&dquo; and therefore who is capable of evaluating a proposed study.4Reviewers may be unable to remain disinterested when research funds are scarce and they are comimport of particular research problems or on the validity of particular techniques.6 Finally, peer review may even be inappropriate when the criterion for judging proposed research is &dquo;utility&dquo;rather than the advancement of &dquo;pure&dquo; scientific knowledge.' In all, the critics argue, claims that decisions on funding are based purely on merit should be viewed skeptically; a researcher's &dquo;track record&dquo; or past performance, for example, is a mandated component of National Science Foundation (NSF) and National Institutes of Health (NIH) peer review. Analysts have also observed how peer decisions could have implicit-if not explicit-political biases.8 These criticisms do not demolish arguments for the practical utility of peer review, but 'they do suggest the necessity for examining its workings and unintended sideeffects.Despite the apparent importance of peer review in influencing funding decisions, and despite growing criticism of alleged deficiencies and biases in peer review systems, not very many empirical studies have been done.9 Most examinations have been limited to addressing a few key questions, either because the study has been contracted for by the funding agency whose system is being studied, or because the study has been conducted &dquo;in-house&dquo; (for example, the &dquo;Kirschstein Report, &dquo;1° which details reforms of NIH peer review).Although there has been no suggestion that these investigators had deliberate biases, various factors extending to data collection, analysis, and interpretation have engendered challenges to the effectiveness of peer review. 11 Moreover, there has

Scientific peer review

“…For the National Institute of Handicapped Research (now National Institute on Disability and Rehabilitation Research, NIDRR), a survey of applicants' opinions found that 41 percent of respondents did not agree with the following statement: “I think that as a whole, the peer reviewer comments were fair” (Fuhrer & Grabois, 1985, p. 319). In a survey of applicants for grants from the National Cancer Institute (NCI), 40 percent of applicants found that reviewers were biased against researchers in minor universities or institutions in certain regions of the U.S. (Gillespie, Chubin, & Kurzon, 1985).…”

“…Further surveys showed similar results regarding the perceived fairness of peer review; these were published by Chubin and Hackett (1990), McCullough (1989, 1994), and Resnik, Gutierrez‐Ford, and Peddada (2008). Predictably there is a close connection between satisfaction with the peer review process and both one's own success in a review and one's participation in a satisfaction survey: Successful applicants and authors are more satisfied and more often take part in surveys than unsuccessful applicants (Gillespie et al, 1985).…”

What is open peer review? A systematic review

“…Social and publication biases: Although often idealized as impartial, objective assessors, in reality studies suggest that peer reviewers may be subject to social biases on the grounds of gender ( Budden et al , 2008; Lloyd, 1990; Tregenza, 2002), nationality ( Daniel, 1993; Ernst & Kienbacher, 1991; Link, 1998), institutional affiliation ( Dall’Aglio, 2006; Gillespie et al , 1985; Peters & Ceci, 1982), language ( Cronin, 2009; Ross et al , 2006; Tregenza, 2002) and discipline ( Travis & Collins, 1991). Other studies suggest so-called “publication bias”, where prejudices against specific categories of works shape what is published.…”

‘Your comments are meaner than your score’: score calibration talk influences intra- and inter-panel variability during scientific grant peer review

“…Such studies have drawn on interviews with various agencies’ staff (Cole, Rubin and Cole 1978; Chubin and Hackett 1990) or with reviewers themselves (Hackett 1987; Lamont 2009); surveys of grant applicants (Gillespie, Chubin and Kurzon 1985; McCullough 1989); quantitative analyses of scores and funding outcomes (Klahr 1985; Sigelman and Scioli 1987; Wenneras and Wold 1997); and de-identified written critiques of funded applications (Porter and Rossini 1985). Such studies rely on post hoc data, with little attention paid to what goes on during review panel meetings; indeed, Olbrecht and Bornmann (2010) identified only five empirical studies that examined the judgment processes of peer review panels.…”