Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data

Abstract

Scientific peer-review and publication systems incur a huge burden in terms of costs and time. Innovative alternatives have been proposed to improve the systems, but assessing their impact in experimental studies is not feasible at a systemic level. We developed an agent-based model by adopting a unified view of peer review and publication systems and calibrating it with empirical journal data in the biomedical and life sciences. We modeled researchers, research manuscripts and scientific journals as agents. Researchers were characterized by their scientific level and resources, manuscripts by their scientific value, and journals by their reputation and acceptance or rejection thresholds. These state variables were used in submodels for various processes such as production of articles, submissions to target journals, in-house and external peer review, and resubmissions. We collected data for a sample of biomedical and life sciences journals regarding acceptance rates, resubmission patterns and total number of published articles. We adjusted submodel parameters so that the agent-based model outputs fit these empirical data. We simulated 105 journals, 25,000 researchers and 410,000 manuscripts over 10 years. A mean of 33,600 articles were published per year; 19 % of submitted manuscripts remained unpublished. The mean acceptance rate was 21 % after external peer review and rejection rate 32 % after in-house review; 15 % publications resulted from the first submission, 47 % the second submission and 20 % the third submission. All decisions in the model were mainly driven by the scientific value, whereas journal targeting and persistence in resubmission defined whether a manuscript would be published or abandoned after one or many rejections. This agent-based model may help in better understanding the determinants of the scientific publication and peer-review systems. It may also help in assessing and identifying the most promising alternative systems of peer review.

The Global Burden of Journal Peer Review in the Biomedical Literature: Strong Imbalance in the Collective Enterprise

“…The evidence base for alternative peer-review systems is still sparse [40, 41]. An evidence-based approach to study peer review, combining computer modeling, experimental studies and sharing of data from journals and publishers, should be encouraged [42–45]. Improvements in peer review will come in response to evidence.…”

Mobile learning on higher educational institutions: how to encourage it?. Simulation approach

“…Normal distributions are wildly used for representing heterogeneity among individuals in agent-based modeling field (for example, see [46][47][48][49]). …”

The Miracle of Peer Review and Development in Science: An Agent-Based Model

“…In models of (Bianchi et al 2017;Kovanis et al 2016), scientists need to allocate time between submissions and reviewing. In our model, both activities are costly if they are performed well and it is not a necessity that scientists invest high effort in one of them.…”

“…The model considers a community of scientists who write and evaluate the papers of each other under different filtering policies of journal editors. Among other agent-based models of peer review, such as Gilbert (1997), Squazzoni and Gandelli (2012), Kovanis et al (2016), Day (2015) and Bianchi et al (2017), the main strength of the current study lies in the fact that it targets the motivational trap of reviewers and highlights that the trap also implies free riding behavior in scientific production. We seek solutions within the model that could potentially guide recommendations for editorial policies to ensure the selection of high quality publications.…”