On Strategyproof Conference Peer Review

Xu, Yichong; Zhao, Han; Shi, Xiaozhong; Zhang, Jeremy; Shah, Nihar B.

doi:10.48550/arxiv.1806.06266

Cited by 9 publications

(17 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We test our algorithms on several real-world datasets. The first real-world dataset is a similarity matrix recreated from ICLR 2018 data in [34]; this dataset has n = 2435 reviewers and d = 911 papers. We also run experiments on similarity matrices created from reviewer bid data for three AI conferences from PrefLib dataset MD-00002 [48], with sizes (n = 31, d = 54), (n = 24, d = 52), and (n = 146, d = 176) respectively.…”

Section: Methodsmentioning

confidence: 99%

“…For all three PrefLib datasets, we transformed "yes," "maybe," and "no response" bids into similarities of 4, 2, and 1 respectively, as is often done in practice [29]. As done in [34], we set loads k = 6 and = 3 for all datasets since these are common loads for computer science conferences (except on the PrefLib2 dataset, for which we set k = 7 for feasibility). All results are averaged over 10 trials with error bars plotted representing the standard error of the mean, although they are sometimes not visible since the variance is very low.…”

Section: Methodsmentioning

confidence: 99%

“…Much prior work has studied the issue of preventing or mitigating strategic behavior in peer review. This work usually focuses on the incentives reviewers have to give poor reviews to other papers in the hopes of increasing their own paper's chances of acceptance [34][35][36][37][38]. Unlike the issues we deal with in this paper, these works consider only reviewers' incentives to get their own paper accepted and not other possible incentives.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments

Jecmen,

Zhang,

Liu

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

We consider three important challenges in conference peer review: (i) reviewers maliciously attempting to get assigned to certain papers to provide positive reviews, possibly as part of quid-pro-quo arrangements with the authors; (ii) "torpedo reviewing," where reviewers deliberately attempt to get assigned to certain papers that they dislike in order to reject them; (iii) reviewer de-anonymization on release of the similarities and the reviewer-assignment code. On the conceptual front, we identify connections between these three problems and present a framework that brings all these challenges under a common umbrella. We then present a (randomized) algorithm for reviewer assignment that can optimally solve the reviewer-assignment problem under any given constraints on the probability of assignment for any reviewer-paper pair. We further consider the problem of restricting the joint probability that certain suspect pairs of reviewers are assigned to certain papers, and show that this problem is NP-hard for arbitrary constraints on these joint probabilities but efficiently solvable for a practical special case. Finally, we experimentally evaluate our algorithms on datasets from past conferences, where we observe that they can limit the chance that any malicious reviewer gets assigned to their desired paper to 50% while producing assignments with over 90% of the total optimal similarity. Our algorithms still achieve this similarity while also preventing reviewers with close associations from being assigned to the same paper.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments

Jecmen,

Zhang,

Liu

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Other proposals involve improvements to various procedural aspects of peer review; these include a recruitment and mentorship pipeline for junior academics to alleviate the scarcity of reviewers (Stelmakh et al, 2020), optimizing reviewer assignment to obtain less noisy reviews (Shah, 2019) and mitigate conflicts of interest (Xu et al, 2018;Jecmen et al, 2020), and better aggregation of subjective opinions on criteria scores to accept/reject recommendations (Noothigattu et al, 2021). Such ideas are complementary to our Stage II reviewing mechanism; they focus on improving paper-reviewer matching or the acceptance decision process, while we focus on incentivizing honest and effortful reviews after reviewers are assigned and before scores are aggregated into accept/reject decisions.…”

Section: Proposals From the Machine Learning Communitymentioning

confidence: 99%

“…However, if the matched reviewers have a conflict of interest, this would weaken the incentive to report honest and effortful reviews. Thus, related work on optimizing paper-reviewer matches (Xu et al, 2018;Jecmen et al, 2020) to avoid conflicts of interest is complementary and important to the success of our mechanism.…”

Section: Conflicts Of Interestmentioning

confidence: 99%

Auctions and Prediction Markets for Scientific Peer Review

Srinivasan¹,

Morgenstern²

2021

Preprint

View full text Add to dashboard Cite

Peer reviewed publications are considered the gold standard in certifying and disseminating ideas that a research community considers valuable. However, we identify two major drawbacks of the current system: (1) the overwhelming demand for reviewers due to a large volume of submissions, and (2) the lack of incentives for reviewers to participate and expend the necessary effort to provide high-quality reviews. In this work, we adopt a mechanism-design approach to propose improvements to the peer review process. We present a two-stage mechanism which ties together the paper submission and review process, simultaneously incentivizing high-quality reviews and high-quality submissions. In the first stage, authors participate in a VCG auction for review slots by submitting their papers along with a bid that represents their expected value for having their paper reviewed. For the second stage, we propose a novel prediction market-style mechanism (H-DIPP) building on recent work in the information elicitation literature, which incentivizes participating reviewers to provide honest and effortful reviews. The revenue raised by the Stage I auction is used in Stage II to pay reviewers based on the quality of their reviews.Preprint. Under review.

show abstract

Optimal Impartial Correspondences

Cembrano

Fischer

Klimm

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We study mechanisms that select a subset of the vertex set of a directed graph in order to maximize the minimum indegree of any selected vertex, subject to an impartiality constraint that the selection of a particular vertex is independent of the outgoing edges of that vertex. For graphs with maximum outdegree d, we give a mechanism that selects at most d + 1 vertices and only selects vertices whose indegree is at least the maximum indegree in the graph minus one. We then show that this is best possible in the sense that no impartial mechanism can only select vertices with maximum degree, even without any restriction on the number of selected vertices. We finally obtain the following trade-off between the maximum number of vertices selected and the minimum indegree of any selected vertex: when selecting at most k vertices out of n, it is possible to only select vertices whose indegree is at least the maximum indegree minus ⌊(n − 2)/(k − 1)⌋ + 1.

show abstract

On Strategyproof Conference Peer Review

Cited by 9 publications

References 31 publications

Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments

Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments

Auctions and Prediction Markets for Scientific Peer Review

Optimal Impartial Correspondences

Contact Info

Product

Resources

About