The Review Process at PSPB: Correlates of Interreviewer Agreement and Manuscript Acceptance

Petty, Richard E.; Fleming, Monique A.; Fabrigar, Leandre R.

doi:10.1177/0146167299025002005

Cited by 53 publications

(36 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In fact, under competitive conditions, the level of agreement between reviewers decreases steadily with each round, such that the average level during the last three rounds was just 0.47. This value is consistent with the empirical level of 0.5 found in the literature (38,39), but significantly lower than the level of consensus under noncompetitive conditions (W = 13,602, P < 0.001). Moreover, under competitive conditions, reviewers are even less consistent than predicted by a null model with shuffled reviews (SI Appendix, Fig.…”

Section: Resultssupporting

confidence: 91%

Peer review and competition in the Art Exhibition Game

Balietti

Goldstone

Helbing

2016

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

To investigate the effect of competitive incentives under peer review, we designed a novel experimental setup called the Art Exhibition Game. We present experimental evidence of how competition introduces both positive and negative effects when creative artifacts are evaluated and selected by peer review. Competition proved to be a double-edged sword: on the one hand, it fosters innovation and product diversity, but on the other hand, it also leads to more unfair reviews and to a lower level of agreement between reviewers. Moreover, an external validation of the quality of peer reviews during the laboratory experiment, based on 23,627 online evaluations on Amazon Mechanical Turk, shows that competition does not significantly increase the level of creativity. Furthermore, the higher rejection rate under competitive conditions does not improve the average quality of published contributions, because more high-quality work is also rejected. Overall, our results could explain why many ground-breaking studies in science end up in lower-tier journals. Differences and similarities between the Art Exhibition Game and scholarly peer review are discussed and the implications for the design of new incentive systems for scientists are explained.peer review | competition | creativity | innovation | fairness C ompetitive incentives are an essential tool to manipulate effort and performance of human groups in many real-life situations (1, 2). Sport tournaments with huge prizes, goal-contingent rewards for employees, and lavish end-of-career bonuses for corporate CEOs are a few examples. However, the literature on incentives and rewards offers mixed evidence of how effective competitive incentives are in improving individual performance (3). In particular, external (monetary) incentives might crowd out intrinsic motivation, which results in no effect, or even a negative effect on individual effort (4-6). Moreover, intrinsic motivation might not only mediate effort, but might actually be necessary to achieve creative performance (7). Similarly, competitive pressure can reduce the performance of professional athletes, causing them to "choke under pressure" (8, 9). Finally, when higher interests are at stake, competition can also directly lead to negative consequences, such as uncooperative behavior and even sabotage (10,11).In this paper, we test the effect of competition in a peer review system. Peer review is a self-regulating system where individuals with similar competence (peers) assess the quality of each other's work. Peer review is widely used by governmental agencies and health care professionals, and it is one of the cornerstones of science. Scholarly peer review is a truly complex system: it involves many actors engaged in multiple roles encompassing various feedback loops (12). Thus far, its inherent complexity and the restricted access to data have made it difficult to investigate peer review. Empirical studies have documented that the review process has low levels of inter-referee agreement (13, 14), lacks reliabilit...

show abstract

Section: Resultssupporting

confidence: 91%

Peer review and competition in the Art Exhibition Game

Balietti

Goldstone

Helbing

2016

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…Third, authors of particular nationalities may produce articles with relatively high impact, potentially on account of preferential patterns of own-country or own-ethnicity citation [GREENWALD & SHUH, 1994;LANGE, 1985]. Fourth, more eminent or productive authors, who have an extensive record of prior accomplishment, may tend to produce more influential articles [BALDI, 1998], just as they tend to receive more positive evaluations from reviewers and editors [PETTY & AL., 1999]. This possibility has received some support from STEWART [1983], who notes that the effect may reflect not only the superior qualities of the cited work, but also the preferential citing of the work because of its author's eminence (the "Matthew effect"; [MERTON, 1968]).…”

Section: Related Studiesmentioning

confidence: 99%

What makes an article influential? Predicting impact in social and personality psychology

et al. 2008

View full text Add to dashboard Cite

“…The second assumption may or may not be the case. It is rare for information about most journal review outcomes to be made public (though see Petty, Fleming, & Fabrigar, 1999). But even if high levels of misreporting Running Head: ANALYTIC REVIEW 20 rates did make news, we suggest that the value of this information is greater than any reputational cost.…”

Section: Analytic Reviews Will Make the Discipline Of Psychology Lookmentioning

confidence: 76%

Analytic Review as a Solution to the Misreporting of Statistical Results in Psychological Science

Sakaluk¹,

Williams²,

Biernat³

2017

Preprint

View full text Add to dashboard Cite

We propose analytic review as a solution to the problem of misreporting statistical results in psychological science. Analytic review requires authors submitting manuscripts for publication to also submit the data file and syntax used during analyses. Regular reviewers or statistical experts then review reported analyses, in order to verify that the analyses reported were actually conducted, and that the statistical values are accurately reported. We begin by describing the problem of misreporting in psychology, and then introduce the basic analytic review process. We then highlight both primary and secondary benefits of adopting analytic review, and describe different permutations of the analytic review system, each with its own strengths and limitations. We conclude by attempting to dispel three anticipated concerns about analytic review, namely: analytic review will increase the workload placed on scholars, analytic review will infringe on the traditional peer-review process, and analytic review will hurt the image of the discipline of psychology. Although implementing analytic review will add one more step to the bureaucratic publication process, we believe it can be implemented in an efficient manner that would greatly assist in decreasing the frequency and impact of misreporting, while also providing secondary benefits in other domains of scientific integrity.

show abstract

The Review Process at PSPB: Correlates of Interreviewer Agreement and Manuscript Acceptance

Cited by 53 publications

References 39 publications

Peer review and competition in the Art Exhibition Game

Peer review and competition in the Art Exhibition Game

What makes an article influential? Predicting impact in social and personality psychology

Analytic Review as a Solution to the Misreporting of Statistical Results in Psychological Science

Contact Info

Product

Resources

About