Peer Review Agreement or Peer Review Disagreement: Which Is Better?

Abstract

Peer review is generally considered the cornerstone of the scientific control system. Hence it is critical that peer review works well. The empirical finding that reviewers often disagree among themselves is the starting point for an analysis of peer review. Depending on the reasons for such disagreements, I argue that disagreement, as well as agreement, among reviewers can have both positive and negative effects for science. The empirical research on peer review is analyzed according to a categorization of review objects (manuscripts or research applications) and review outcomes (approval or rejection) and draws on psychological judgment and decision-making research. Moreover, bias in peer review is scrutinized. The conclusion offers implications for the peer review system and for scientific research.

Communicating and compromising on disciplinary expertise in the peer review of research proposals

“…'Peer consensus' is often believed to be an indicator of 'inter-rater reliability', and is typically regarded as the most valuable collective product of panel deliberation (see Brenneis, 1994;Cicchetti, 1991;Hemlin, 2009;Marsh et al, 2008). It indeed results in a clear signal for funding decisions.…”

Valorizing SSH research: Towards a new approach to evaluate SSH research’ value for society

“…While standards of quality are controversial in all disciplines, recent research shows that perceptions and conceptualisation of excellence are even more complex and fuzzy in the SSH (see, e.g., Furlong & Oancea, 2005;Hemlin, 1996;Williams & Galleron, 2016). Also, while peer-review is generally universally acclaimed and accepted within this area, in many journals or publishing houses, as well as at other levels and institutions where evaluation is practiced by peers, procedures are far from being transparent and robust, and often have not been closely monitored or assessed against principles such as thoroughness and fairness (Hemlin, 2009). Moreover, the relationship between science and society is changing and evaluation mechanisms are bound to reflect that to a certain extent.…”