A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

Kang, Dongyeop; Ammar, Waleed; Dalvi, Bhavana; Zuylen, Madeleine van; Kohlmeier, Sebastian; Hovy, Eduard; Schwartz, Roy

doi:10.18653/v1/n18-1149

Cited by 131 publications

(168 citation statements)

References 13 publications

Supporting

Mentioning

164

Contrasting

Order By: Relevance

“…a paper is, the less confident the reviewers will be. Note that our correlation results are different from those reported by Kang et al (2018), who report that the OVAL has low Pearson correlation with SND (0.01) and ORG (0.08). While the differences might be caused by a variation in aspect definitions, we believe that our estimate is more reliable as the dataset analyzed in Kang et al (2018) is substantially smaller than ours.…”

Section: Reviewscontrasting

confidence: 99%

Does My Rebuttal Matter? Insights from a Major

Gao¹,

Eger²,

Kuznetsov³

et al. 2019

Proceedings of the 2019 Conference of the North

View full text Add to dashboard Cite

Peer review is a core element of the scientific process, particularly in conference-centered fields such as ML and NLP. However, only few studies have evaluated its properties empirically. Aiming to fill this gap, we present a corpus that contains over 4k reviews and 1.2k author responses from ACL-2018. We quantitatively and qualitatively assess the corpus. This includes a pilot study on paper weaknesses given by reviewers and on quality of author responses. We then focus on the role of the rebuttal phase, and propose a novel task to predict after-rebuttal (i.e., final) scores from initial reviews and author responses. Although author responses do have a marginal (and statistically significant) influence on the final scores, especially for borderline papers, our results suggest that a reviewer's final score is largely determined by her initial score and the distance to the other reviewers' initial scores. In this context, we discuss the conformity bias inherent to peer reviewing, a bias that has largely been overlooked in previous research. We hope our analyses will help better assess the usefulness of the rebuttal phase in NLP conferences.

show abstract

Section: Reviewscontrasting

confidence: 99%

Does My Rebuttal Matter? Insights from a Major

Gao¹,

Eger²,

Kuznetsov³

et al. 2019

Proceedings of the 2019 Conference of the North

View full text Add to dashboard Cite

show abstract

“…For Task 2, we observe that the handcrafted feature-based system by Kang et al (2018) performs inferior compared to the baselines. This is because the features were very naive and did not 4 https://github.com/aritzzz/DeepSentiPeer address the complexity involved in such a task.…”

Section: Results and Analysismentioning

confidence: 97%

“…For Task 1, we can see that our review sentiment augmented approach outperforms the baselines and the comparing systems by a wide margin (∼ 29% reduction in error) on the ICLR 2017 dataset. With only using review+sentiment information, we are still able to outperform Kang et al (2018) by a margin of 11% in terms of RMSE. A further relative error reduction of 19% with the addition of paper features strongly suggests that only review is not sufficient for the final recommendation.…”

Section: Methodsmentioning

confidence: 94%

“…For the decision prediction task, the comparing system performs even worse, and we outperform them by a considerable margin of 28% (ACL 2017) and 26% (CoNLL 2017), respectively ( Table 3). The reason is that the work reported in Kang et al (2018) relies on elementary handcrafted features extracted only from the paper; does not consider the review features whereas we include the review features along with the sentiment information in our deep neural architecture. However, we also find that our approach with only Review+Sentiment performs inferior to the Paper+Review method in Kang et al (2018) for ACL 2017.…”

Section: Cross-domain Experimentsmentioning

confidence: 99%

See 1 more Smart Citation

DeepSentiPeer: Harnessing Sentiment in Review Texts to Recommend Peer Review Decisions

Ghosal¹,

Verma²,

Ekbal³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Automatically validating a research artefact is one of the frontiers in Artificial Intelligence (AI) that directly brings it close to competing with human intellect and intuition. Although criticized sometimes, the existing peer review system still stands as the benchmark of research validation. The present-day peer review process is not straightforward and demands profound domain knowledge, expertise, and intelligence of human reviewer(s), which is somewhat elusive with the current state of AI. However, the peer review texts, which contains rich sentiment information of the reviewer, reflecting his/her overall attitude towards the research in the paper, could be a valuable entity to predict the acceptance or rejection of the manuscript under consideration. Here in this work, we investigate the role of reviewers sentiments embedded within peer review texts to predict the peer review outcome. Our proposed deep neural architecture takes into account three channels of information: the paper, the corresponding reviews, and the review polarity to predict the overall recommendation score as well as the final decision. We achieve significant performance improvement over the baselines (∼ 29% error reduction) proposed in a recently released dataset of peer reviews. An AI of this kind could assist the editors/program chairs as an additional layer of confidence in the final decision making, especially when non-responding/missing reviewers are frequent in present day peer review.

show abstract

“…We collect review data from three sources: (1) openreview.net-an online peer reviewing platform for ICLR 2017, ICLR 2018, and UAI 2018 2 ; (2) reviews released for accepted papers at NeurIPS from 2013 to 2017; and (3) opted-in reviews for ACL 2017 from Kang et al (2018).…”

Section: Ampere Datasetmentioning

confidence: 99%

Argument Mining for Understanding Peer Reviews

Hua¹,

Nikolov²,

Badugu³

et al. 2019

Proceedings of the 2019 Conference of the North

View full text Add to dashboard Cite

Peer-review plays a critical role in the scientific writing and publication ecosystem. To assess the efficiency and efficacy of the reviewing process, one essential element is to understand and evaluate the reviews themselves.In this work, we study the content and structure of peer reviews under the argument mining framework, through automatically detecting (1) argumentative propositions put forward by reviewers, and (2) their types (e.g., evaluating the work or making suggestions for improvement). We first collect 14.2K reviews from major machine learning and natural language processing venues. 400 reviews are annotated with 10, 386 propositions and corresponding types of EVALUATION, REQUEST, FACT, REFERENCE, or QUOTE. We then train state-of-the-art proposition segmentation and classification models on the data to evaluate their utilities and identify new challenges for this new domain, motivating future directions for argument mining. Further experiments show that proposition usage varies across venues in amount, type, and topic.

show abstract

A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

Cited by 131 publications

References 13 publications

Does My Rebuttal Matter? Insights from a Major

Does My Rebuttal Matter? Insights from a Major

DeepSentiPeer: Harnessing Sentiment in Review Texts to Recommend Peer Review Decisions

Argument Mining for Understanding Peer Reviews

Contact Info

Product

Resources

About