Matching Papers and Reviewers at Large Conferences

Leyton‐Brown, Kevin; Mausam, Mausam; Nandwani, Yatin; Zarkoob, Hedayat; Cameron, Chris; Newman, N.B.; Raghu, Dinesh

doi:10.48550/arxiv.2202.12273

Cited by 4 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many automated peer review systems maximize a sum of point-wise, ad-hoc affinity scores. Charlin and Zemel [2013] construct affinity scores using either LDA topic-modeling or a predictive model based on bag of words text representations, while Leyton-Brown et al [2022] compute affinity scores as a fixed function of keyword matching, reviewer bids, and TPMS and ACL document similarity scores. Other objectives are often used as well; Payan and Zick [2022] approximately maximize affinity scores subject to a fairness constraint, while other works maximize the minimum score assigned to any given paper or group of papers [Kobren et al, 2019, Stelmakh et al, 2019, Aziz et al, 2023.…”

Section: Related Workmentioning

confidence: 99%

“…As an example of how this information is combined, affinity scores are currently implemented in OpenReview as a linear combination of the TPMS scores and the reviewer bids. 1 Recent conferences such as AAAI'22 and IJCAI'22 took a similar approach, linearly combining TPMS scores, ACL scores, and SAM scores and raising the sum to an exponential power based on the reviewer bids [Leyton-Brown et al, 2022].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Into the Unknown: Assigning Reviewers to Papers with Uncertain Affinities

Cousins¹,

Payan²,

Zick³

2023

Preprint

View full text Add to dashboard Cite

Peer review cannot work unless qualified and interested reviewers are assigned to each paper. Nearly all automated reviewer assignment approaches estimate real-valued affinity scores for each paper-reviewer pair that act as proxies for the predicted quality of a future review; conferences then assign reviewers to maximize the sum of these values. This procedure does not account for noise in affinity score computationreviewers can only bid on a small number of papers, and textual similarity models are inherently probabilistic estimators. In this work, we assume paper-reviewer affinity scores are estimated using a probabilistic model. Using these probabilistic estimates, we bound the scores with high probability and maximize the worst-case sum of scores for a reviewer allocation. Although we do not directly recommend any particular method for estimation of probabilistic affinity scores, we demonstrate how to robustly maximize the sum of scores across multiple different models. Our general approach can be used to integrate a large variety of probabilistic paper-reviewer affinity models into reviewer assignment, opening the door to a much more robust peer review process.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Into the Unknown: Assigning Reviewers to Papers with Uncertain Affinities

Cousins¹,

Payan²,

Zick³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The problem of malicious bid manipulation has been taken seriously by recent conferences, who have attempted a variety of approaches to address it. For example, AAAI 2021 implemented several techniques to break up colluding reviewers, including preventing two-cycles in the reviewer assignment and requiring geographic diversity among the reviewers assigned to the same paper [12]. They also required a minimum number of positive bids from each reviewer, with the aim of preventing reviewers from targeting a specific paper.…”

Section: Related Workmentioning

confidence: 99%

“…These bids are often given a high weight in the similarity computation, allowing each reviewer to have a significant level of control over their own assignment. For example, at the AAAI 2021 conference [12]: "Reviewers were assigned papers for which they bid positively (willing or eager) 77.4% of the time. A back-ofthe-envelope calculation leads us to estimate that 79.3% of these matches may not have happened had the reviewer not bid positively."…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Dataset on Malicious Paper Bidding in Peer Review

Steven¹,

Yoon²,

Conitzer³

et al. 2022

Preprint

View full text Add to dashboard Cite

In conference peer review, reviewers are often asked to provide "bids" on each submitted paper that express their interest in reviewing that paper. A paper assignment algorithm then uses these bids (along with other data) to compute a high-quality assignment of reviewers to papers. However, this process has been exploited by malicious reviewers who strategically bid in order to unethically manipulate the paper assignment, crucially undermining the peer review process. For example, these reviewers may aim to get assigned to a friend's paper as part of a quid-pro-quo deal. A critical impediment towards creating and evaluating methods to mitigate this issue is the lack of any publicly-available data on malicious paper bidding. In this work, we collect and publicly release a novel dataset to fill this gap, collected from a mock conference activity where participants were instructed to bid either honestly or maliciously. We further provide a descriptive analysis of the bidding behavior, including our categorization of different strategies employed by participants. Finally, we evaluate the ability of each strategy to manipulate the assignment, and also evaluate the performance of some simple algorithms meant to detect malicious bidding. The performance of these detection algorithms can be taken as a baseline for future research on detecting malicious bidding.

show abstract

A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences

Stelmakh

Shah

Singh

et al. 2021

AAAI

View full text Add to dashboard Cite

Conference peer review constitutes a human-computation process whose importance cannot be overstated: not only it identifies the best submissions for acceptance, but, ultimately, it impacts the future of the whole research area by promoting some ideas and restraining others. A surge in the number of submissions received by leading AI conferences has challenged the sustainability of the review process by increasing the burden on the pool of qualified reviewers which is growing at a much slower rate. In this work, we consider the problem of reviewer recruiting with a focus on the scarcity of qualified reviewers in large conferences. Specifically, we design a procedure for (i) recruiting reviewers from the population not typically covered by major conferences and (ii) guiding them through the reviewing pipeline. In conjunction with the ICML 2020 --- a large, top-tier machine learning conference --- we recruit a small set of reviewers through our procedure and compare their performance with the general population of ICML reviewers. Our experiment reveals that a combination of the recruiting and guiding mechanisms allows for a principled enhancement of the reviewer pool and results in reviews of superior quality compared to the conventional pool of reviews as evaluated by senior members of the program committee (meta-reviewers).

show abstract

Matching Papers and Reviewers at Large Conferences

Cited by 4 publications

References 0 publications

Into the Unknown: Assigning Reviewers to Papers with Uncertain Affinities

Into the Unknown: Assigning Reviewers to Papers with Uncertain Affinities

A Dataset on Malicious Paper Bidding in Peer Review

A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences

Contact Info

Product

Resources

About