Making peer review evidence-based: It’s time to open the "black box"

Abstract

Peer review serves an essential role in the cultivation, validation, and dissemination of social work knowledge and scholarship. Nevertheless, the current peer review system has many limitations. It is charged as being unreliable, biased, ineffective, and unaccountable, among numerous other issues. That said, peer review is still commonly viewed as the best possible system of knowledge governance, given the relevant alternatives. In this research note, I scrutinize this assumption. Although peer review can sometimes be effective, it is not therefore a rigorous or even dependable system. Indeed, the practice of peer review in social work is overwhelmingly closed and opaque, and assurances of its rigor are speculative at best. Given that social work research informs policies and practices that have real world consequences for clients and communities, it is imperative that our research – and its appraisal – be held to the strictest of standards. This includes our system(s) of peer review. After highlighting common criticisms of traditional peer review, I articulate a research agenda on “open peer review” which can reform how peer review is performed, provide feedback to editors and reviewers, and help make the process more rigorous, transparent, and evidence-based. Implications for social work education are explored and discussed.

On the Dearth of Retractions in Social Work: A Preliminary Analysis of Ten Leading Journals

“…Though recent developments in scholarly publishing have enabled the partial or even complete decoupling of peer review from this gatekeeping role (sometimes referred to as "journal-independent" or "journal agnostic" peer review; e.g., Eisen et al, 2022;Hamelin et al, 2022;Lumb 2023), social work journals still largely adhere to a traditional model of double-blind, pre-publication peer review (Caputo, 2019). While the published literature is in some ways validated by this model, the underlying processes are largely unstandardized and opaque, and its overall functioning is poorly understood (see Dunleavy, 2022b;Tennant & Ross-Hellauer, 2019). Blatant errors (e.g., misreported or incorrect statistical results; inaccurate or misleading citations), omissions (e.g., selective reporting of results), misrepresentations (inflation or deflation of findings), and even cases of fraud (e.g., fabrication of data, falsification) are inevitably published (see Dunleavy & Lacasse, 2023;Ioannidis, 2005Ioannidis, , 2008Nuijten et al, 2015;Srivastava, 2016, and references therein).…”

Re%3A++Mandatory+data+and+code+sharing+for+research+published+by+The+BMJ

“…(1) I and many others in the open science community share the belief that making research, scholarship, and its appraisal (e.g., peer review) open and transparent will help improve the rigor, accountability, and trustworthiness across all levels of the research lifecycle. (2,3,4) This mandatory sharing policy could be enhanced by making explicit and public a plan for evaluating its effectiveness. At some predetermined time (e.g., 1 May 2025, 31, December 2025), and at set-intervals thereafter (e.g., every 3-, 5-years), it would be useful to see published: what percentage of clinical trials published in The BMJ actually do have publicly available data and code, the extent to which this can be accessed unimpeded (a concern shared by other readers, especially those outside of academic and research institutions), and perhaps most importantly the extent to which the data and code are able to be used to reproduce and replicate findings.…”