Peer Review as an Evolving Response to Organizational Constraint: Evidence from Sociology Journals, 1952–2018

Merriman, Ben

doi:10.1007/s12108-020-09473-x

Cited by 13 publications

(19 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We discuss three bodies of related work: 1) proposals to improve scientific peer review from the machine learning community, 2) general proposals to reform or replace scientific peer review, and 3) the information elicitation without verification literature (Faltings and Radanovic, 2017;Waggoner and Chen, 2013). Readers may also be interested in a broader historical overview of the development of peer review (focused primarily on the social sciences) presented by Merriman (2020).…”

Section: Related Workmentioning

confidence: 99%

“…Yet, despite its widespread use and acceptance, it is plagued by a number of well-documented problems: willing reviewers are often in short supply and reviews themselves are often of poor quality, giving rise to noisy acceptance decisions (Lawrence and Cortes, 2015;Shah et al, 2018). Studying the history of peer review in sociology, Merriman (2020) writes that while peer review is 'a practice intended to assess the merits of pieces of scholarship, its features did not arise because they were believed to be an especially apt means of identifying the best work.' The issues with peer review have become especially salient in machine learning as well as computing more broadly.…”

Section: Introductionmentioning

confidence: 99%

“…With this context, we now discuss the often cited primary motivations for and criticisms of peer review below. Merriman (2020) argues that these 'intellectual' motivations tend to be post-hoc rationalizations, and the current process of peer review is 'best understood as the product of continuous efforts to steward editors' scarce attention while preserving an open submission policy that favors authors' interests'. Part of our goal then is to develop a peer review mechanism that lives up to the stated 'intellectual' motivations for the process.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Auctions and Prediction Markets for Scientific Peer Review

Srinivasan¹,

Morgenstern²

2021

Preprint

View full text Add to dashboard Cite

Peer reviewed publications are considered the gold standard in certifying and disseminating ideas that a research community considers valuable. However, we identify two major drawbacks of the current system: (1) the overwhelming demand for reviewers due to a large volume of submissions, and (2) the lack of incentives for reviewers to participate and expend the necessary effort to provide high-quality reviews. In this work, we adopt a mechanism-design approach to propose improvements to the peer review process. We present a two-stage mechanism which ties together the paper submission and review process, simultaneously incentivizing high-quality reviews and high-quality submissions. In the first stage, authors participate in a VCG auction for review slots by submitting their papers along with a bid that represents their expected value for having their paper reviewed. For the second stage, we propose a novel prediction market-style mechanism (H-DIPP) building on recent work in the information elicitation literature, which incentivizes participating reviewers to provide honest and effortful reviews. The revenue raised by the Stage I auction is used in Stage II to pay reviewers based on the quality of their reviews.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Auctions and Prediction Markets for Scientific Peer Review

Srinivasan¹,

Morgenstern²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…They showed how the community-based editorial process at the Royal Society struggled to keep pace with the growth of science. Merriman (2021) analyzed when major elements of peer review emerged in journals of the American Sociological Association and argued that the ongoing evolution of peer review in these journals has not been driven by epistemic considerations but rather by 'efforts to steward the scarce attention of editors while preserving an open submission policy that favors the authors' interests' (Merriman, 2021, p. 341).…”

Section: Some Theoretical Efforts On Peer Reviewmentioning

confidence: 99%

Towards Theorizing Peer Review

Hug¹

2021

Preprint

View full text Add to dashboard Cite

Academic peer review is seriously undertheorized because peer review studies focus on discovering and confirming phenomena, such as biases, and are much less concerned with explaining, predicting, or controlling phenomena on a theoretical basis. In this paper, I therefore advocate for more theorizing in research on peer review. I first describe the main characteristics of the peer review literature, and, based on this, I argue why theory is useful in research on peer review. I then present some theoretical efforts on peer review and conclude by encouraging peer review researchers to be more theoretically engaged. This invitation to theory building complements recent roadmaps and calls that have emphasized that we need better access to peer review data, improve research design and statistical analysis in peer review studies, experiment with innovative approaches to peer review, and provide more funding for peer review research.

show abstract

“…An additional factor that may explain the difference between articles and peer reviews is that at least traditionally, peer reviews were not solicited to give authors comments that would improve their work. Rather, peer review in the 20th century originated as advice to editors about whether to accept or reject a manuscript and/or whether to invite a revision (Burnham, 1990; Merriman, 2021). While some consider ad hominem comments to be inappropriate even when they have that purpose, one could argue that the capabilities of the authors are relevant to whether it is worthwhile to invite a revision.…”

Section: Ad Hominems Not Expressed In Journal Articles But Condoned B...mentioning

confidence: 99%

Ad hominem rhetoric in scientific psychology

Holcombe

2021

British J of Psychology

View full text Add to dashboard Cite

Ad hominem discourse is largely prohibited in scientific journals. Historically, this prohibition restricted the dissemination of ad hominem discussion, but during the last decade, blogs and social media platforms became popular among researchers. With the use of social media now entrenched among researchers, there are important questions about the role of ad hominems.Ad hominems and other forms of strong criticism became particularly evident in online discussions associated with the recent replication crisis in psychology. Here, these discussions, and a few incidences of ad hominems in journal articles, are situated in the broader history of science. It is argued that explicit codes of conduct should be considered to curb certain kinds of ad hominem comments in certain fora, but that some ad hominem discussions have an important role to play in a healthier science.

show abstract

Peer Review as an Evolving Response to Organizational Constraint: Evidence from Sociology Journals, 1952–2018

Cited by 13 publications

References 30 publications

Auctions and Prediction Markets for Scientific Peer Review

Auctions and Prediction Markets for Scientific Peer Review

Towards Theorizing Peer Review

Ad hominem rhetoric in scientific psychology

Contact Info

Product

Resources

About