Peer Review: A System under Stress

Gropp, Robert E.; Glisson, Scott R.; Gallo, Stephen A.; Thompson, Lisa

doi:10.1093/biosci/bix034

“…However, while the number of contributors is very large, more than 30 million, one third of all edits are made by only 10,000 people, just 0.03% ( wikipedia.org/wiki/Wikipedia:List_of_Wikipedians_by_number_of_edits ). This is broadly similar to what is observed in current academic peer review systems, where the majority of the work is performed by a minority of the participants ( Fox et al , 2017 ; Gropp et al , 2017 ; Kovanis et al , 2016 ).…”

Section: Potential Future Modelssupporting

confidence: 81%

A multi-disciplinary perspective on emergent and future innovations in peer review

Tennant

¹

,

Dugan

²

,

Graziotin

³

et al. 2017

View full text Add to dashboard Cite

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

show abstract

“…However, while the number of contributors is very large, more than 30 million, one third of all edits are made by only 10,000 people, just 0.03% ( wikipedia.org/wiki/Wikipedia:List_of_Wikipedians_by_number_of_edits). This is broadly similar to what is observed in current academic peer review systems, where the majority of the work is performed by a minority of the participants ( Fox et al , 2017; Gropp et al , 2017; Kovanis et al , 2016). …”

Section: Potential Future Modelssupporting

confidence: 81%

“…By incentivizing peer review, much of its potential burden can be alleviated by widening the potential referee pool concomitant with the growth in review requests. This can also help to diversify the process and inject transparency into peer review, a solution that is especially appealing when considering that it is often a small minority of researchers who perform the vast majority of peer reviews ( Fox et al , 2017; Gropp et al , 2017); for example, in biomedical research, only 20 percent of researchers perform 70–95 percent of the reviews ( Kovanis et al , 2016). In 2014, a working group on peer review services (CASRAI) was established to “develop recommendations for data fields, descriptors, persistence, resolution, and citation, and describe options for linking peer-review activities with a person identifier such as ORCID ” ( Paglione & Lawrence, 2015).…”

Section: The Traits and Trends Affecting Modern Peer Reviewmentioning

confidence: 99%

A multi-disciplinary perspective on emergent and future innovations in peer review

Dugan

¹

,

Graziotin

²

,

Jacques

³

et al. 2017

View full text Add to dashboard Cite

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

show abstract

“…This creates an incredible burden on the global research workforce, considering that a typical research paper requires 2–3 referees and a handling editor, most of whom act on a volunteer basis for scholarly journals. This has created a state commonly referred to as ‘reviewer fatigue’ (Breuning et al 2015 ; C. W. Fox, Albert and Vines 2017 ), and available evidence suggests that the majority of reviews are performed by a minority of researchers within an increasingly over-burdened system (Lyman 2013 ; Jubb 2016 ; Gropp et al 2017 ; Kovanis et al 2017 ; J. Fox and Petchey 2010 ; Vines, Rieseberg and Smith 2010 ).…”

Section: The Present State Of Scholarly Peer Reviewmentioning

confidence: 99%

The state of the art in peer review

Tennant¹

2018

FEMS Microbiology Letters

View full text Add to dashboard Cite

Scholarly communication is in a perpetual state of disruption. Within this, peer review of research articles remains an essential part of the formal publication process, distinguishing it from virtually all other modes of communication. In the last several years, there has been an explosive wave of innovation in peer review research, platforms, discussions, tools and services. This is largely coupled with the ongoing and parallel evolution of scholarly communication as it adapts to rapidly changing environments, within what is widely considered as the ‘open research’ or ‘open science’ movement. Here, we summarise the current ebb and flow around changes to peer review and consider its role in a modern digital research and communications infrastructure and suggest why uptake of new models of peer review appears to have been so low compared to what is often viewed as the ‘traditional’ method of peer review. Finally, we offer some insight into the potential futures of scholarly peer review and consider what impacts this might have on the broader scholarly research ecosystem. In particular, we focus on the key traits of certification and reputation, moderation and quality control and engagement incentives, and discuss how these interact with socio-technical aspects of peer review and academic culture.

show abstract

Peer Review: A System under Stress

Cited by 45 publications

References 6 publications

A multi-disciplinary perspective on emergent and future innovations in peer review

A multi-disciplinary perspective on emergent and future innovations in peer review

A multi-disciplinary perspective on emergent and future innovations in peer review

The state of the art in peer review

Contact Info

Product

Resources

About