Peer Grading the Peer Reviews: A Dual-Role Approach for Lightening the Scholarly Paper Review Process

Arous, Ines; Yang, Jie; Khayati, Mourad; Cudré-Mauroux, Philippe

doi:10.1145/3442381.3450088

Cited by 5 publications

(11 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To evaluate the Rcon (see next section), we need to grade these reviews. Arous et al (2021) have conducted a large-scale crowdsourcing study for review grading, in which they asked participants to provide ratings for each of the eight conformity criteria for each of the reviews. However, they only selected a subset of the most informative reviews from the ICLR 2018 (30% reviews) and 2019 (5% reviews) data sets for grading.…”

Section: Data Setmentioning

confidence: 99%

Assessing and predicting the quality of peer reviews: a text mining approach

Meng

2023

View full text Add to dashboard Cite

Purpose This paper aims to quantify the quality of peer reviews, evaluate them from different perspectives and develop a model to predict the review quality. In addition, this paper investigates effective features to distinguish the reviews' quality. Design/methodology/approach First, a fine-grained data set including peer review data, citations and review conformity scores was constructed. Second, metrics were proposed to evaluate the quality of peer reviews from three aspects. Third, five categories of features were proposed in terms of reviews, submissions and responses using natural language processing (NLP) techniques. Finally, different machine learning models were applied to predict the review quality, and feature analysis was performed to understand effective features. Findings The analysis results revealed that reviewers become more conservative and the review quality becomes worse over time in terms of these indicators. Among the three models, random forest model achieves the best performance on all three tasks. Sentiment polarity, review length, response length and readability are important factors that distinguish peer reviews’ quality, which can help meta-reviewers value more worthy reviews when making final decisions. Originality/value This study provides a new perspective for assessing review quality. Another originality of the research lies in the proposal of a novelty task that predict review quality. To address this task, a novel model was proposed which incorporated various of feature sets, thereby deepening the understanding of peer reviews.

show abstract

Section: Data Setmentioning

confidence: 99%

Assessing and predicting the quality of peer reviews: a text mining approach

Meng

2023

View full text Add to dashboard Cite

show abstract

“…David Tran et al quantified reproducibility in the review process by Monte-Carlo simulations [2]. Ines Arous et al proposed a Bayesian framework that integrates a machine learning model with peer grading to assess the conformity of scholarly reviews [5]. To evaluate the generated review, Yuan et al proposed a variety of diagnostic criteria for review quality, including review aspect coverage and informativeness [4].…”

Section: B Review Assessmentmentioning

confidence: 99%

“…Apart from that, peer review has also been challenged by the rapid increase of paper submissions. Consider the example of computer science conferences: The Conference on Neural Information Processing Systems (NeurIPS) received 9467 submissions in 2020, which is five times the number of submissions it received in 2010 [5] . The exploding nature of paper submissions leads to paper-vetting by less-experienced researchers from disparate fields due to the shortage of qualified reviewers.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

NeurReview: A Neural Architecture Based Conformity Prediction of Peer Reviews

Meng

2023

IEEE Access

View full text Add to dashboard Cite

Peer review is at the heart of scholarly communications and the foundation of scientific evaluation. However, peer review's effectiveness is continuously challenged due to biased and inconsistent peer reviews. Consequently, ensuring the quality of peer reviews is a time-critical problem. In this paper, we investigate the conformity between reviews and meta-reviews. To predict the review conformity and identify the effective features to distinguish the misaligned reviews, we propose NeurReview, which models the review process from the review structure and interactions with authors and other reviewers. Two evaluation datasets are constructed from the ICLR open reviews. The evaluation results verified the efficacy of our proposed model. In addition, we found that the divergence with other reviews and responses, the consistency of sentiment polarity with the recommendation score, etc., are beneficial features for identifying lowconformity reviews, which can assist meta-reviewers in making final decisions.

show abstract

“…There are different patterns of human-machine collaboration for solving real-world problems (Kamar, Hacker, and Horvitz 2012;Russakovsky, Li, and Fei-Fei 2015;Chandra et al 2020;Arous et al 2021) while reducing human worker efforts (Yi et al 2012). A typical approach is to use AI programs to identify data items that require attention from human workers (Nguyen, Wallace, and Lease 2015;Yang et al 2019;Wilder, Horvitz, and Kamar 2020;Liu et al 2020).…”

Section: Related Workmentioning

confidence: 99%

Human+AI Crowd Task Assignment Considering Result Quality Requirements

Kobayashi

Wakabayashi

Morishima

2021

HCOMP

View full text Add to dashboard Cite

This paper addresses the problem of dynamically assigning tasks to a crowd consisting of AI and human workers. Currently, crowdsourcing the creation of AI programs is a common practice. To apply such kinds of AI programs to the set of tasks, we often take the ``all-or-nothing'' approach that waits for the AI to be good enough. However, this approach may prevent us from exploiting the answers provided by the AI until the process is completed, and also prevents the exploration of different AI candidates. Therefore, integrating the created AI, both with other AIs and human computation, to obtain a more efficient human-AI team is not trivial. In this paper, we propose a method that addresses these issues by adopting a ``divide-and-conquer'' strategy for AI worker evaluation. Here, the assignment is optimal when the number of task assignments to humans is minimal, as long as the final results satisfy a given quality requirement. This paper presents some theoretical analyses of the proposed method and an extensive set of experiments conducted with open benchmarks and real-world datasets. The results show that the algorithm can assign many more tasks than the baselines to AI when it is difficult for AIs to satisfy the quality requirement for the whole set of tasks. They also show that it can flexibly change the number of tasks assigned to multiple AI workers in accordance with the performance of the available AI workers.

show abstract

Peer Grading the Peer Reviews: A Dual-Role Approach for Lightening the Scholarly Paper Review Process

Cited by 5 publications

References 40 publications

Assessing and predicting the quality of peer reviews: a text mining approach

Assessing and predicting the quality of peer reviews: a text mining approach

NeurReview: A Neural Architecture Based Conformity Prediction of Peer Reviews

Human+AI Crowd Task Assignment Considering Result Quality Requirements

Contact Info

Product

Resources

About