A scoping review of simulation models of peer review

Abstract

Peer review is a process used in the selection of manuscripts for journal publication and proposals for research grant funding. Though widely used, peer review is not without flaws and critics. Performing large-scale experiments to evaluate and test correctives and alternatives is difficult, if not impossible. Thus, many researchers have turned to simulation studies to overcome these difficulties. In the last 10 years this field of research has grown significantly but with only limited attempts to integrate disparate models or build on previous work. Thus, the resulting body of literature consists of a large variety of models, hinging on incompatible assumptions, which have not been compared, and whose predictions have rarely been empirically tested. This scoping review is an attempt to understand the current state of simulation studies of peer review. Based on 46 articles identified through literature searching, we develop a proposed taxonomy of model features that include model type (e.g. formal models vs. ABMs or other) and the type of modeled peer review system (e.g. peer review in grants vs. in journals or other). We classify the models by their features (including some core assumptions) to help distinguish between the modeling approaches. Finally, we summarize the models’ findings around six general themes: decision-making, matching submissions/reviewers, editorial strategies; reviewer behaviors, comparisons of alternative peer review systems, and the identification and addressing of biases. We conclude with some open challenges and promising avenues for future modeling work.

Analyzing sentiments in peer review reports: Evidence from two science funding agencies

“…Comparative empirical research across systems has been especially difficult. Simulation research on peer review has been a promising approach where the lack of empirical data is hindering progress (Roebber, & Schultz, 2011;Feliciani, Luo, et al, 2019). Automated methods for supporting journal peer review have been suggested as far back as the 1980s (Garfield, 1987).…”

Towards Theorizing Peer Review

“…Murray et al (2019) examined whether peer review outcomes at a biosciences journal are influenced by author-referee homophily (i.e., the preference of referees for manuscripts from authors with shared -10 -characteristics) and found evidence for higher rates of acceptance in the case of gender and country homophily. In a scoping review, Feliciani et al (2019) identified 46 studies that represented elements of peer review in computational models (e.g., agent-based models, latent Markov models). Computational models are thus likely the most common approach so far to theorize peer review.…”

Capacity Building in Peer-Review: a Student-led Journal Experience

“…The Committee on Publication Ethics (COPE) asserts that peer-review in all its forms plays a pivotal role in ensuring the integrity and liability of scholarly records (1), which leads to a higher quality of research. However, several criticisms of the peer-review process exist (2).…”

“…The current knowledge dissemination model is characterized by the presentation of research ndings which go through the inspection of journals that de ne priority themes, formats, and speci c evidence. In this regard, journals are able to decide what is published, by whom, when, and how (2,5,6). Because of this structure, such a model does not encourage controversy or public debate.…”