Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators’ Expertise

Dhull, Komal; Steven, Jecmen,; Kothari, Pravesh K.; Shah, Nihar B.

doi:10.1609/hcomp.v10i1.21987

Cited by 6 publications

(3 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In a recent work, Srinivasan and Morgenstern (2021) propose the idea of using the VCG mechanism to elicit bids from authors and using peer prediction mechanisms to evaluate reviews and reward the reviewers (with virtual money). In dealing with the malicious bidding problem, a stream of literature focuses on designing and optimizing the paper-reviewer assignment mechanism to guarantee strategyproofness (Aziz et al 2019;Jecmen et al 2020;Dhull et al 2022;Xu et al 2018).…”

Section: Related Workmentioning

confidence: 99%

Eliciting Honest Information from Authors Using Sequential Review

Zhang,

Schoenebeck,

2024

AAAI

View full text Add to dashboard Cite

In the setting of conference peer review, the conference aims to accept high-quality papers and reject low-quality papers based on noisy review scores. A recent work proposes the isotonic mechanism, which can elicit the ranking of paper qualities from an author with multiple submissions to help improve the conference's decisions. However, the isotonic mechanism relies on the assumption that the author's utility is both an increasing and a convex function with respect to the review score, which is often violated in realistic settings (e.g.~when authors aim to maximize the number of accepted papers). In this paper, we propose a sequential review mechanism that can truthfully elicit the ranking information from authors while only assuming the agent's utility is increasing with respect to the true quality of her accepted papers. The key idea is to review the papers of an author in a sequence based on the provided ranking and conditioning the review of the next paper on the review scores of the previous papers. Advantages of the sequential review mechanism include: 1) eliciting truthful ranking information in a more realistic setting than prior work; 2) reducing the reviewing workload and increasing the average quality of papers being reviewed; 3) incentivizing authors to write fewer papers of higher quality.

show abstract

Section: Related Workmentioning

confidence: 99%

Eliciting Honest Information from Authors Using Sequential Review

Zhang,

Schoenebeck,

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Thus, a number of works has explored the idea of assignment fairness, aiming at producing more balanced assignments (Kobren et al, 2019;Stelmakh et al, 2021). Finally, other works explore the ideas of envy-freeness (Tan et al, 2021;Payan, 2022), resistance to lone-wolf strategic behavior (Xu et al, 2019;Dhull et al, 2022), and encouraging various types of diversity (Li et al, 2015;Leyton-Brown et al, 2022).…”

Section: Evaluation Of Similarity-computation Algorithmsmentioning

confidence: 99%

A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences

Stelmakh

Shah

Singh

et al. 2021

AAAI

View full text Add to dashboard Cite

Conference peer review constitutes a human-computation process whose importance cannot be overstated: not only it identifies the best submissions for acceptance, but, ultimately, it impacts the future of the whole research area by promoting some ideas and restraining others. A surge in the number of submissions received by leading AI conferences has challenged the sustainability of the review process by increasing the burden on the pool of qualified reviewers which is growing at a much slower rate. In this work, we consider the problem of reviewer recruiting with a focus on the scarcity of qualified reviewers in large conferences. Specifically, we design a procedure for (i) recruiting reviewers from the population not typically covered by major conferences and (ii) guiding them through the reviewing pipeline. In conjunction with the ICML 2020 --- a large, top-tier machine learning conference --- we recruit a small set of reviewers through our procedure and compare their performance with the general population of ICML reviewers. Our experiment reveals that a combination of the recruiting and guiding mechanisms allows for a principled enhancement of the reviewer pool and results in reviews of superior quality compared to the conventional pool of reviews as evaluated by senior members of the program committee (meta-reviewers).

show abstract

“…These mechanisms incentivize coordination on similar evaluation reports by punishing evaluations that do not match each other. Thus, they do not necessarily incentivize accuracy (Prelec, 2004;Miller et al, 2005;Jurca and Faltings, 2009;Faltings et al, 2012;Dasgupta and Ghosh, 2013;Lev et al, 2023;Waggoner and Chen, 2014;Shnayder et al, 2016;Dhull et al, 2022). Any such mechanism introduces uninformative equilibria alongside the truth-telling one (Jurca and Faltings, 2009;Waggoner and Chen, 2014).…”

Section: Introductionmentioning

confidence: 99%

Removing Bias and Incentivizing Precision in Peer-grading

Chakraborty,

Jindal,

Nath

2024

jair

View full text Add to dashboard Cite

Most peer-evaluation practices rely on the evaluator’s goodwill and model them as potentially noisy evaluators. But what if graders are competitive, i.e., enjoy higher utility when their peers get lower scores? We model the setting as a multi-agent incentive design problem and propose a new mechanism, PEQA, that incentivizes these agents (peer-graders) through a score-assignment rule and a grading performance score. PEQA is designed in such a way that it makes grader-bias irrelevant and ensures grader-utility to be monotonically increasing with the grading-precision, despite competitiveness. When grading is costly and costs are private information of the individual graders, a modified version of PEQA implements the socially optimal grading-choices in equilibrium. Data from our classroom experiments is consistent with our theoretical assumptions and show that PEQA outperforms the popular median mechanism, which is used in several massive open online courses (MOOCs).

show abstract

Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators’ Expertise

Cited by 6 publications

References 36 publications

Eliciting Honest Information from Authors Using Sequential Review

Eliciting Honest Information from Authors Using Sequential Review

A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences

Removing Bias and Incentivizing Precision in Peer-grading

Contact Info

Product

Resources

About