A multi-disciplinary perspective on emergent and future innovations in peer review

Abstract

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

Managing the Growth of Peer Review at the Royal Society Journals, 1865-1965

“…The historical example of learned society publishing, including that of the Royal Society, has long been used to support calls for academics to take back the control of research journals (Harnad 1995;Fyfe et al 2017;Tennant et al 2017). Looking at the past should provoke reflection among those seeking to use digital communications technologies to create virtual communities aiming to self-organize editorial work, refereeing, and publication (e.g., Tennant et al 2017), in order to recreate collegial dialogue and judgment (Hirschauer 2010), rather than distant trilateral negotiations between authors, referees, and editors (Myers 1985). Questions remain about how best to organize those communities and how to motivate members to carry out voluntary work, while keeping the pace of the complexity of the editorial work and increasing requests for responsibility and accountability (Fitzpatrick 2010).…”

“…According to recent estimates, about 2.5 million articles are published each year in academic journals across the world, with numbers growing about 3 percent a year (Ware and Mabe 2015;Publons 2018). At the heart of the contemporary system of academic publishing lies the process of peer review, which has become a central mechanism in establishing the credibility of scholarly journals and the reliability of scientific knowledge claims (Siler and Strang 2017;Tennant et al 2017;Moxham and Fyfe 2018;Grimaldo, Marušić, and Squazzoni 2018;Horbach and Halffman 2018). The growing prominence of peer review in advising both editorial and grant-making decisions has led to increasing scrutiny of the process in recent decades.…”

“…Thanks to the growth of international research communities and digital communications technologies, editors can rely upon a large pool of anonymous, distant, and internationally dispersed peer reviewers who are personally unknown to each other, and have little in common but their shared research expertise and standards (Lamont 2009). The apparent loss of a sense of community commitment, and of group accountability in academic judgment, has become a theme in current debates on the evolution of scholarly communication Torny 2014, 2015;Tennant et al 2017). Analyzing the editorial implications of the link between journals and the society that organized them encourages us to reflect on pros and cons of the community (dis)embeddedness of modern scholarly journals (Kean 2009).…”

Navigating Diverse Data Science Learning: Critical Reflections Towards Future Practice

“…Plenty of practical tips are supplied regarding things that are not covered by the syllabus such as collaboration and peer review tools (e.g. GitHub, Jupyter [24], and other generic [25] and fieldspecific tools [26], [27]) and best practices regarding project management and presentation strategies that are suitable for academia and industry.…”

Can editors save peer review from peer reviewers?

“…Many of those ideas have in turn not been definitively shown to outperform peer review [16], and are met with skepticism from proponents of classical peer review [17, 18]. (For overviews of the history and the current debate on peer review, see [10, 18–20]).…”