Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments

Abstract

We consider three important challenges in conference peer review: (i) reviewers maliciously attempting to get assigned to certain papers to provide positive reviews, possibly as part of quid-pro-quo arrangements with the authors; (ii) "torpedo reviewing," where reviewers deliberately attempt to get assigned to certain papers that they dislike in order to reject them; (iii) reviewer de-anonymization on release of the similarities and the reviewer-assignment code. On the conceptual front, we identify connections between these three problems and present a framework that brings all these challenges under a common umbrella. We then present a (randomized) algorithm for reviewer assignment that can optimally solve the reviewer-assignment problem under any given constraints on the probability of assignment for any reviewer-paper pair. We further consider the problem of restricting the joint probability that certain suspect pairs of reviewers are assigned to certain papers, and show that this problem is NP-hard for arbitrary constraints on these joint probabilities but efficiently solvable for a practical special case. Finally, we experimentally evaluate our algorithms on datasets from past conferences, where we observe that they can limit the chance that any malicious reviewer gets assigned to their desired paper to 50% while producing assignments with over 90% of the total optimal similarity. Our algorithms still achieve this similarity while also preventing reviewers with close associations from being assigned to the same paper.

Reviewer Assignment Problem: A Systematic Review of the Literature

“…The manual information-based approach assigns reviewers to proposals using additional information provided by reviewers. Bidding, also known as rating, is the most prevalent strategy in manual-based systems, in which the reviewer is involved in the process of allocating their preferences (Benferhat & Lang, 2001;Basu et al, 2001;Janak et al, 2005;Goldsmith & Sloan, 2007;Papagelis & Plexousakis, 2008;Xu et al, 2010;Garg et al, 2010;Kolasa & Krol, 2011;Chen et al, 2012;Daş & Gökçen, 2014;Tayal et al, 2014;Liu et al, 2015;Liu et al, 2016;Yeşilçimen & Yıldırım, 2019;Jecmen et al, 2020;Fiez et al, 2020). In the bidding method, reviewers must review the abstracts or the full text of the proposals and assign scores for each proposal representing their review willingness.…”

“…The last but most important disadvantage is that these approaches are open to malicious behaviors and conflicts of interest that will affect the fairness of the evaluation. With these approaches, reviewers can significantly increase their chances of being assigned to a proposal they can target by bidding strategically (Jecmen et al, 2020;Wu et al, 2021).…”

“…These three works examined only the reviewers' motivations to have their own proposals accepted. Jecmen et al (2020) thought of other possible motivations for a reviewer to provide untruthful reviews, such as a dislike for the research field or disincentives caused by applicant-reviewer collaboration. Jecmen et al (2020) proposed randomized reviewer-paper assignments by optimizing the sum of similarity but with some randomness to prevent torpedo reviewing, untruthful favorable reviews, and reviewer reviews deanonymization.…”

Can We Automate Scientific Reviewing?

“…However, at the same time it is easy to identify many issues: expensiveness, slowness, existence of inconsistency (Langford & Guzdial, 2015) and bias (Tomkins et al, 2017), etc. Some efforts have been put into analyzing the peer review process including automating review assignment (Jin et al, 2017;Nguyen et al, 2018;Anjum et al, 2019;Jecmen et al, 2020), examining bias problems (Tomkins et al, 2017;Stelmakh et al, 2019), examining consistency problems (Langford & Guzdial, 2015) and performing sentiment analysis on reviews (Wang & Wan, 2018;Chakraborty et al, 2020). Several decision classification methods have been explored to help make accept or reject decision given a paper.…”

Can We Automate Scientific Reviewing?

“…However, at the same time it is easy to identify many issues: expensiveness, slowness, existence of inconsistency (Langford and Guzdial, 2015) and bias (Tomkins et al, 2017), etc. Some efforts have been put into analyzing the peer review process including automating review assignment (Jin et al, 2017;Nguyen et al, 2018;Anjum et al, 2019;Jecmen et al, 2020), examining bias problems (Tomkins et al, 2017;Stelmakh et al, 2019), examining consistency problems (Langford and Guzdial, 2015) and performing sentiment analysis on reviews (Wang and Wan, 2018;Chakraborty et al, 2020). Several decision classification methods have been explored to help make accept or reject decision given a paper.…”