Detecting Problem Statements in Peer Assessments

Xiao, Yunkai; Zingle, Gabriel; Jia, Qinjin; Shah, Harsh; Zhang, Yi; Li, Tianyi; Karovaliya, Mohsin; Zhao, Weixiang; Song, Yang; Ji, Jie; Balasubramaniam, Ashwin; Harshit, Patel; Bhalasubbramanian, Priyankha; Patel, Vikram; Gehringer, Edward F.

doi:10.48550/arxiv.2006.04532

Cited by 1 publication

(2 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zingle et al compared rule-based machine-learning and deep neural-network methods for detecting suggestions in peer assessments, and the result showed that deep-learning methods outperformed other traditional methods [37]. Xiao et al collected around 20,000 peer-review comments and leveraged different neural networks to detect problems in peer assessments [31].…”

Section: Related Work 21 Automated Peer-review Evaluationmentioning

confidence: 99%

“…Zingle et al [37] utilized different rule-based, machine-learning, and deep-learning methods for detecting suggestions in peer-review comments. However, to the best of our knowledge, no single study exists that investigates using a multi-task learning (MTL) model to detect multiple features simultaneously (as illustrated in Figure 1), albeit extensive research has been carried out on the topic of automated peer-review evaluation (e.g., [34,32,33,31,30,37,13,19,8]).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

ALL-IN-ONE: Multi-Task Learning BERT models for Evaluating Peer Assessments

Jia¹,

Cui²,

Xiao³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Peer assessment has been widely applied across diverse academic fields over the last few decades, and has demonstrated its effectiveness. However, the advantages of peer assessment can only be achieved with high-quality peer reviews. Previous studies have found that high-quality review comments usually comprise several features (e.g., contain suggestions, mention problems, use a positive tone). Thus, researchers have attempted to evaluate peer-review comments by detecting different features using various machine learning and deep learning models. However, there is no single study that investigates using a multi-task learning (MTL) model to detect multiple features simultaneously. This paper presents two MTL models for evaluating peer-review comments by leveraging the state-of-the-art pre-trained language representation models BERT and DistilBERT. Our results demonstrate that BERT-based models significantly outperform previous GloVe-based methods by around 6% in F1-score on tasks of detecting a single feature, and MTL further improves performance while reducing model size.

show abstract

Section: Related Work 21 Automated Peer-review Evaluationmentioning

confidence: 99%