Reputation or peer review? The role of outliers

Abstract

We present an agent-based model of paper publication and consumption that allows to study the effect of two different evaluation mechanisms, peer review and reputation, on the quality of the manuscripts accessed by a scientific community. The model was empirically calibrated on two data sets, mono- and multi-disciplinary. Our results point out that disciplinary settings differ in the rapidity with which they deal with extreme events—papers that have an extremely high quality, that we call outliers. In the mono-disciplinary case, reputation is better than traditional peer review to optimize the quality of papers read by researchers. In the multi-disciplinary case, if the quality landscape is relatively flat, a reputation system also performs better. In the presence of outliers, peer review is more effective. Our simulation suggests that a reputation system could perform better than peer review as a scientific information filter for quality except when research is multi-disciplinary and in a field where outliers exist.Electronic supplementary materialThe online version of this article (10.1007/s11192-018-2826-3) contains supplementary material, which is available to authorized users.

A scoping review of simulation models of peer review

“…With the exception of the two root models, most peer review models were not further developed after their first publication. This observation echoes the conclusion by Grimaldo et al (2018, 2018) about the state of the broader literature on peer review, where fragmentation and lack of collaboration and/or knowledge sharing also prevails.…”

“…Grimaldo and co-authors developed two changes to a standard peer review system: reviewer accountability, also called ‘disagreement control’ (Grimaldo and Paolucci 2012, 2013; Grimaldo et al 2012), and a reputation system (Grimaldo et al 2018b). The idea of reviewer accountability hinges on the notion that repeated disagreement between reviewers may be a signal of poor reviews, or of selfish reviewer behavior.…”

Agent-based model: A method worthy of promotion in Library and Information Science

“…Simulating the cooperative game, they evaluated the impact of different resource allocations on peer review, and then analysed the impact of policy-making on peer review. Grimaldo et al [36] used ABM to simulate the behaviour of journals receiving highquality articles, and especially discussed the influence of outliers on peer review and journal word-of-mouth. Garcı ´a's and Chamorro-Padial's team [37][38][39] published papers three times, analysing how editors' preferences affect contributors in 2015, explaining why assassins and zealots evolutionary appear in peer review in 2019, and discussing the shortcomings of peer review incentive system in 2020.…”

Reading Peer Review

“…Aiming, with substantial European research funding, to 'improve [the] efficiency, transparency and accountability of peer review through a trans-disciplinary, cross-sectorial collaboration', the consortium has been one of the most prolific centres for research into peer review in the past half decade. Publications from the group have spanned the author perspective on peer review (Drvenica et al 2019), the reward systems of peer review (Zaharie and Seeber 2018), the links between reputation and peer review (Grimaldo, Paolucci, and Sabater-Mir 2018), the role that artificial intelligence might play in future structures of review (Mrowinski et al 2017), the timescales involved in review (Huisman and Smits 2017;Mrowinski et al 2016), the reasons why people cite retracted papers (Bar-Ilan and Halevi 2017), the fate of rejected manuscripts (Casnici, Grimaldo, Gilbert, Dondio et al 2017), and the ways in which referees act in multidisciplinary contexts .…”