HirePeer: Impartial Peer-Assessed Hiring at Scale in Expert Crowdsourcing Markets

Kotturi, Yasmine; Kahng, Anson; Procaccia, Ariel D.; Kulkarni, Chinmay

doi:10.1609/aaai.v34i03.5641

Cited by 10 publications

(14 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This work falls in the line of empirical works that study various behavioral aspects of human computation, including motivational aspect (Kaufmann et al, 2011) and the impact of the task framing on performance (Kotturi et al, 2020;Levy and Sarne, 2018;Chandler and Kapelner, 2013). The findings we report in this paper can be combined with insights from the aforementioned works to improve the design of the review process with a goal of achieving better efficiency and engagement of reviewers.…”

Section: Introductionmentioning

confidence: 61%

A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences

Stelmakh¹,

Shah²,

Singh³

et al. 2020

Preprint

View full text Add to dashboard Cite

Conference peer review constitutes a human-computation process whose importance cannot be overstated: not only it identifies the best submissions for acceptance, but, ultimately, it impacts the future of the whole research area by promoting some ideas and restraining others. A surge in the number of submissions received by leading AI conferences has challenged the sustainability of the review process by increasing the burden on the pool of qualified reviewers which is growing at a much slower rate. In this work, we consider the problem of reviewer recruiting with a focus on the scarcity of qualified reviewers in large conferences. Specifically, we design a procedure for (i) recruiting reviewers from the population not typically covered by major conferences and (ii) guiding them through the reviewing pipeline. In conjunction with ICML 2020 -a large, top-tier machine learning conference -we recruit a small set of reviewers through our procedure and compare their performance with the general population of ICML reviewers. Our experiment reveals that a combination of the recruiting and guiding mechanisms allows for a principled enhancement of the reviewer pool and results in reviews of superior quality compared to the conventional pool of reviews as evaluated by senior members of the program committee (meta-reviewers).

show abstract

Section: Introductionmentioning

confidence: 61%

A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences

Stelmakh¹,

Shah²,

Singh³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Research on peer assessment. This paper also falls in the line of several recent works in computer science on the peer-evaluation process that includes both empirical (Tomkins et al, 2017;Sajjadi et al, 2016;Kotturi et al, 2020) and theoretical (Wang and Shah, 2018;Stelmakh et al, 2018;Noothigattu et al, 2018;Fiez et al, 2020) studies. Particularly relevant works are recent papers (Tomkins et al, 2017;Stelmakh et al, 2019) that consider the problem of detecting biases (e.g., gender bias) in single-blind peer review.…”

Section: Related Literaturementioning

confidence: 73%

“…This approach, however, has two limitations. First, past work suggests that the test and control rankings may have different distributions even under the absence of manipulations due to misalignment of incentives (Kotturi et al, 2020). Second, existing works (Mania et al, 2018;Gretton et al, 2012;Jiao and Vert, 2018;Rastogi et al, 2020) on two-sample testing with rankings ignore the authorship information that is crucial in our case as we show in the sequel (Section 3.2).…”

Section: Related Literaturementioning

confidence: 85%

“…Peer-assessment techniques offer an appealing alternative: instead of relying on independent judges, they distribute the evaluation task across the fellow applicants and then aggregate the received reviews into the final ranking of items. This paradigm has become popular for employee evaluation (Edwards and Ewen, 1996) and grading students' homeworks (Topping, 1998), and is now expanding to more novel applications of massive open online courses (Kulkarni et al, 2013;Piech et al, 2013) and hiring at scale (Kotturi et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

“…While impartial methods remove the benefits of manipulations, such robustness may come at the cost of some accuracy loss when reviewers do not engage in strategic behaviour. This loss is caused by less efficient data usage (Kahng et al, 2018;Xu et al, 2019) and reduction of efforts put by reviewers (Kotturi et al, 2020). Implementation of such methods also introduces some additional logistical burden on the system designers; as a result, in many critical applications (e.g., conference peer review) the non-impartial mechanisms are employed.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment

Stelmakh¹,

Shah²,

Singh³

2020

Preprint

View full text Add to dashboard Cite

We consider the issue of strategic behaviour in various peer-assessment tasks, including peer grading of exams or homeworks and peer review in hiring or promotions. When a peer-assessment task is competitive (e.g., when students are graded on a curve), agents may be incentivized to misreport evaluations in order to improve their own final standing. Our focus is on designing methods for detection of such manipulations. Specifically, we consider a setting in which agents evaluate a subset of their peers and output rankings that are later aggregated to form a final ordering. In this paper, we investigate a statistical framework for this problem and design a principled test for detecting strategic behaviour. We prove that our test has strong false alarm guarantees and evaluate its detection ability in practical settings. For this, we design and execute an experiment that elicits strategic behaviour from subjects and release a dataset of patterns of strategic behaviour that may be of independent interest. We then use the collected data to conduct a series of real and semi-synthetic evaluations that demonstrate a strong detection power of our test.

show abstract

Bias-aware ranking from pairwise comparisons

Ferrara,

Bonchi,

Fabbri

et al. 2024

Data Min Knowl Disc

View full text Add to dashboard Cite

Human feedback is often used, either directly or indirectly, as input to algorithmic decision making. However, humans are biased: if the algorithm that takes as input the human feedback does not control for potential biases, this might result in biased algorithmic decision making, which can have a tangible impact on people’s lives. In this paper, we study how to detect and correct for evaluators’ bias in the task of ranking people (or items) from pairwise comparisons. Specifically, we assume we are given pairwise comparisons of the items to be ranked produced by a set of evaluators. While the pairwise assessments of the evaluators should reflect to a certain extent the latent (unobservable) true quality scores of the items, they might be affected by each evaluator’s own bias against, or in favor, of some groups of items. By detecting and amending evaluators’ biases, we aim to produce a ranking of the items that is, as much as possible, in accordance with the ranking one would produce by having access to the latent quality scores. Our proposal is a novel method that extends the classic Bradley-Terry model by having a bias parameter for each evaluator which distorts the true quality score of each item, depending on the group the item belongs to. Thanks to the simplicity of the model, we are able to write explicitly its log-likelihood w.r.t. the parameters (i.e., items’ latent scores and evaluators’ bias) and optimize by means of the alternating approach. Our experiments on synthetic and real-world data confirm that our method is able to reconstruct the bias of each single evaluator extremely well and thus to outperform several non-trivial competitors in the task of producing a ranking which is as much as possible close to the unbiased ranking.

show abstract

HirePeer: Impartial Peer-Assessed Hiring at Scale in Expert Crowdsourcing Markets

Cited by 10 publications

References 32 publications

A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences

A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences

Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment

Bias-aware ranking from pairwise comparisons

Contact Info

Product

Resources

About