How to Calibrate the Scores of Biased Reviewers by Quadratic Programming

Abstract

Peer reviewing is the key ingredient of evaluating the quality of scientific work. Based on the review scores assigned by the individual reviewers to the submissions, program committees of conferences and journal editors decide which papers to accept for publication and which to reject. However, some reviewers may be more rigorous than others, they may be biased one way or the other, and they often have highly subjective preferences over the papers they review. Moreover, each reviewer usually has only a very local view, as he or she evaluates only a small fraction of the submissions. Despite all these shortcomings, the review scores obtained need to be aggregrated in order to globally rank all submissions and to make the acceptance/rejection decision. A common method is to simply take the average of each submission's review scores, possibly weighted by the reviewers' confidence levels. Unfortunately, the global ranking thus produced often suffers a certain unfairness, as the reviewers' biases and limitations are not taken into account. We propose a method for calibrating the scores of reviewers that are potentially biased and blindfolded by having only partial information. Our method uses a maximum likelihood estimator, which estimates both the bias of each individual reviewer and the unknown "ideal" score of each submission. This yields a quadratic program whose solution transforms the individual review scores into calibrated, globally comparable scores. We argue why our method results in a fairer and more reasonable global ranking than simply taking the average of scores. To show its usefulness, we test our method empirically using real-world data.

Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators’ Expertise

“…Finally, there are various other types of strategic or dishonest behavior in peer assessment Hvistendahl 2013;Ferguson, Marcus, and Oransky 2014;Fanelli 2009;Resnik, Gutierrez-Ford, and Peddada 2008;Vijaykumar 2020;Littman 2021;Jecmen et al 2020;Wu et al 2021) and the design of computational methods to mitigate such behavior is vital. More generally, peer assessment is an important application with a broad set of challenges including subjectivity (Lee 2015;Noothigattu, Shah, and Procaccia 2021), miscalibration (Roos, Rothe, and Scheuermann 2011;Wang and Shah 2019), biases (Tomkins, Zhang, and Heavlin 2017;Stelmakh, Shah, and Singh 2019a;Manzoor and Shah 2021), and others (Meir et al 2020;Fiez, Shah, and Ratliff 2020;Wang et al 2021;Shah 2021).…”

How do authors’ perceptions of their papers compare with co-authors’ perceptions and peer-review decisions?

“…We show the results in Fig 5 . Observe that interestingly, among the pairs where both authors gave a strict ranking, they disagreed 32% This high amount of disagreement between co-authors about the scientific contribution of their jointly authored papers has some implications for research on peer review. Many models of peer review [21,38,[46][47][48][49][50] assume existence of some "true quality" of each paper. This result raises questions about such an assumption-if there were such a true quality, then it is perhaps the authors who would know them well at least in a relative sense, but as we saw above, authors do not seem to agree.…”

What have we learned from OpenReview?

“…Mrowinski et al [17] exploit evolutionary computation to improve editorial strategies in peer review. Roos et al [18] propose a method for calibrating the ratings of potentially biased reviewers via a maximum likelihood estimation (MLE) approach. Stelmakh et al [19] discuss biases due to demographics in single-blind peer review and study associated hypothesis testing problems.…”