Measuring the effectiveness of scientific gatekeeping

Abstract

Peer review is the main institution responsible for the evaluation and gestation of scientific research. Although peer review is widely seen as vital to scientific evaluation, anecdotal evidence abounds of gatekeeping mistakes in leading journals, such as rejecting seminal contributions or accepting mediocre submissions. Systematic evidence regarding the effectiveness-or lack thereof-of scientific gatekeeping is scant, largely because access to rejected manuscripts from journals is rarely available. Using a dataset of 1,008 manuscripts submitted to three elite medical journals, we show differences in citation outcomes for articles that received different appraisals from editors and peer reviewers. Among rejected articles, desk-rejected manuscripts, deemed as unworthy of peer review by editors, received fewer citations than those sent for peer review. Among both rejected and accepted articles, manuscripts with lower scores from peer reviewers received relatively fewer citations when they were eventually published. However, hindsight reveals numerous questionable gatekeeping decisions. Of the 808 eventually published articles in our dataset, our three focal journals rejected many highly cited manuscripts, including the 14 most popular; roughly the top 2 percent. Of those 14 articles, 12 were deskrejected. This finding raises concerns regarding whether peer review is ill-suited to recognize and gestate the most impactful ideas and research. Despite this finding, results show that in our case studies, on the whole, there was value added in peer review. Editors and peer reviewers generally-but not always-made good decisions regarding the identification and promotion of quality in scientific manuscripts.peer review | innovation | decision making | publishing | creativity P eer review alters science via the filtering out of rejected manuscripts and the revision of eventually published articles. Publication in leading journals is linked to professional rewards in science, which influences the choices scientists make with their work (1). Although peer review is widely cited as central to academic evaluation (2, 3), numerous scholars have expressed concern about the effectiveness of peer review, particularly regarding the tendency to protect the scientific status quo and suppress innovative findings (4, 5). Others have focused on errors of omission in peer review, offering anecdotes of seminal scientific innovations that faced emphatic rejections from high-status gatekeepers and journals before eventually achieving publication and positive regard (6-8). Unfortunately, systematic study of peer review is difficult, largely because of the sensitive and confidential nature of the subject matter. Based on a dataset of 1,008 manuscripts submitted to three leading medical journals-Annals of Internal Medicine, British Medical Journal, and The Lancet-we analyzed the effectiveness of peer review. In our dataset, 946 submissions were rejected and 62 were accepted. Among the rejections, we identified 757 manuscripts eventually publis...

“Excellence R Us”: university research and the fetishisation of excellence

“…These suggest a bias towards conformance and social connectedness over innovation in funding decisions in a world in which success rates are as low as 10%. It also provides further evidence of funding-agency bias against disruptively innovative work noted by many researchers over the years (Kuhn [1962] 2012; Campanario, 1993Campanario, , 1995Campanario, , 1996Campanario, , 2009Costello, 2010;Ioannidis et al, 2014;Siler et al, 2015).…”

“…Campanario, Gans and Shepherd, and others, for example, have traced the rejection histories of Nobel and other prize winners, including for papers reporting on results for which they later won their recognition (Gans and Shepherd, 1994;Campanario, 2009;Azoulay et al, 2011: 527-528). Campanario and others have also reported on the initial rejection of papers that later went on to become among the more highly cited in their fields or in the journals that ultimately accepted them (Campanario, 1993(Campanario, , 1996Campanario, 1995;Campanario and Acedo, 2007;Calcagno et al, 2012;Nicholson and Ioannidis, 2012;Siler et al, 2015). Yet others have found a generally poor relationship between high ratings in grant competitions and subsequent "productivity" as measured by publication or citation counts (Pagano, 2006;Costello, 2010;Lindner and Nakamura, 2015;Fang et al, 2016;Meng, 2016).…”

A multi-disciplinary perspective on emergent and future innovations in peer review

“…A popular editorial in The BMJ made some quiter serious allegations at peer review, stating that it is “slow, expensive, profligate of academic time, highly subjective, prone to bias, easily abused, poor at detecting gross defects, and almost useless at detecting fraud” ( Smith, 2006). In addition, beyond editorials, a substantial corpus of studies has now critically examined the technical aspects of conventional journal article peer review (e.g., ( Armstrong, 1997; Bruce et al , 2016; Jefferson et al , 2007; Overbeke, 1999; Pöschl, 2012; Siler et al , 2015a)), with overlapping and some times contrasting results.…”

“…Attempts to reproduce how peer review selects what is worthy of publication demonstrate that the process is generally adequate for detecting reliable research, but often fails to recognize the research that has the greatest impact ( Mahoney, 1977; Moore et al , 2017; Siler et al , 2015b). Many critics now view traditional peer review as sub-optimal and detrimental to research because it causes publication delays, with repercussions on the dissemination of novel research ( Armstrong, 1997; Bornmann & Daniel, 2010a; Brembs, 2015; Eisen, 2011; Jubb, 2016; Vines, 2015b).…”

“…This makes it impossible to trace the discussions that led to (sometimes substantial) revisions to the original research ( Bedeian, 2003), the decision process leading to the final publication, or whether peer review even took place. By operating as a closed system, it protects the status quo and suppresses research viewed as radical, innovative, or contrary to the theoretical or established perspectives of referees ( Alvesson & Sandberg, 2014; Benda & Engels, 2011; Horrobin, 1990; Mahoney, 1977; Merton, 1968; Siler et al , 2015a; Siler & Strang, 2017), even though it is precisely these factors that underpin and advance research. As a consequence, questions arise as to the competency, effectiveness, and integrity, as well as participatory elements, of traditional peer review, such as: who are the gatekeepers and how are the gates constructed; what is the balance between author-reviewer-editor tensions and how are these power relations and conflicts resolved; what are the inherent biases associated with this; does this enable a fair or structurally inclined system of peer review to exist; and what are the repercussions for this on our knowledge generation and communication systems?…”