Improving peer review with CARMA

Abstract

We propose to test and improve the quality of peer review through CARMA (Community‐based Assessment of Review Materials): an online repository of review materials that will include all intermediate versions of published manuscripts, associated reviews, and author replies to reviewer comments. CARMA would add the content of reviews to the public scientific record without compromising the anonymity of reviewers. The benefits to the relevant stakeholders are as follows: • For readers, CARMA would reveal the actual impact of peer review on scholarly manuscripts; • CARMA would encourage reviewers to provide thorough evaluations without jeopardizing their candor; • Editors could utilize CARMA for quality assurance by soliciting ratings of reviews from their readership; • For publishers, implementing CARMA would incur minimal costs, and could increase revenues. Whatever the present quality of peer review, making this process more transparent would be in the best interest of meritocratic scholars and society at large.

Exploring the Peer Review Process: What is it, Does it Work, and Can it Be Improved?

“…The open (or unmasked) approach is said to provide an ethical advantage (Godlee 2002), in that those passing judgment must be accountable to the author for their comments and evaluation, plus gaining credit for their contributions. Though this approach could lead to more thorough, rigorous reviews, given that the reviewer's reputation to the author is at stake (Carmi and Koch 2007), even under a single-blind system, reviewers are clearly accountable to the editor (Davidoff 1998). Criticisms include the possibility of payback for either positive or negative reviews in the future, given that reviewers and authors will probably interact through their careers, thus reducing the rigor of critical assessments.…”

“…In this regard, it is somewhat similar to the approach now used in AFS journals, save for being double-blinded and limiting revisions to one. Other approaches take advantage of technological advantages of the Internet, including an open moderated approach (where manuscripts are posted for a moderated comment period prior to submission for regular peer review; Gura 2002); an interactive peer review process followed by public discussion (Poschl et al 2004); posting of all intermediate versions of a manuscript, associated reviews, and author responses to the reviewer comments (Carmi and Koch 2007); an Internet collaboration approach (in Scientific American; Waldrop 2008); and a post-publication filtering approach (the Faculty of 1000; Wets et al 2003). None of these approaches is used widely, and no conclusions can currently be drawn as to their effectiveness.…”

Write when hot — submit when not: seasonal bias in peer review or acceptance?

“…Sometimes other factors influence the decision of whether or not a paper is pub-lished. [4][5][6] For example, the reviewers may be biased if they recognize the authors of a given paper or hold an opposing ideological framework. 3,7 But even more seemingly arbitrary factors might come into play.…”

Aggregating post-publication peer reviews and ratings

“…There is an increasing awareness of the problems of the current scientific publication system, which is based on an outdated paradigm, resulted from the constraints of physical space in printed journals, and which largely ignores the possibilities opened by current internet technologies. There also is an increasing interest in alternatives to this paradigm (Greenbaum et al, 2003 ; Van de Sompel et al, 2004 ; Rodriguez et al, 2006 ; Carmi and Coch, 2007 ; Easton, 2007 ; Kriegeskorte, 2009 ; Chang and Aernoudts, 2010 ). Several papers within this journal's Research Topic on Beyond open access present convincingly a vision of a future where the scientific journal's functions are decoupled and/or the pre-publication reviews by about two or three reviewers is replaced or complemented by an ongoing post-publication process of transparent peer review and rating of papers (Kravitz and Baker, 2011 ; Lee, 2011 ; Ghosh et al, 2012 ; Priem and Hemminger, 2012 ; Sandewall, 2012 ; Wicherts et al, 2012 ; Zimmermann et al, 2012 ).…”