DeepReviewer: Collaborative Grammar and Innovation Neural Network for Automatic Paper Review

Leng, Youfang; Yu, Li; Xiong, Jie

doi:10.1145/3340555.3353766

Cited by 12 publications

(5 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They use bi-directional LSTM for sentences, , which prevents parallelization. Leng et al [13] let model learning the semantic, grammar, and innovative features of an article by three main well-designed components simultaneously. However, only sentences within a fixed-size receptive field can interact with each other.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A Modular Hierarchical Model for Paper Quality Evaluation

Deng¹,

Li²,

Yu³

et al. 2023

Artificial Intelligence Advances

View full text Add to dashboard Cite

Paper quality evaluation is of great significance as it helps to select high quality papers from the massive amount of academic papers. However, existing models needs improvement on the interaction and aggregation of the hierarchical structure. These models also ignore the guiding role of the title and abstract in the paper text. To address above two issues, we propose a well-designed modular hierarchical model (MHM) for paper quality evaluation. Firstly, the input to our model is most of the paper text, and no additional information is needed. Secondly, we fully exploit the inherent hierarchy of the text with three encoders with attention mechanisms: a word-to-sentence(WtoS) encoder, a sentence-to-paragraph(StoP) encoder, and a paper encoder. Specifically, the WtoS encoder uses the pre-trained language model SciBERT to obtain the sentence representation from the word representation. The StoP encoder lets sentences in the same paragraph interact and aggregates them to get paragraph embeddings based on importance scores. The paper encoder does interaction among different hierarchical structures of three modules of a paper text: the paper title, abstract sentences, and body paragraphs. Then this encoder aggregates new representations generated into a compact vector. In addition, the paper encoder models the guiding role of the title and abstract, respectively, generating another two compact vectors. We concatenate the above three compact vectors and additional four manual features to obtain the paper representation. This representation is then fed into a classifier to obtain the acceptance decision, which is a proxy for papersÃ¢â‚¬â„¢ quality. Experimental results on a large-scale dataset built by ourselves show that our model consistently outperforms the previous strong baselines in four evaluation metrics. Quantitative and qualitative analyses further validate the superiority of our model.

show abstract

Section: Related Workmentioning

confidence: 99%

“…There exists interaction and aggregation in the hierarchical structure. However, previous works [10][11][12][13][14][15] does not fully consider the position and contextual information of elements in interaction, and elements' importance in aggregation. Meanwhile they operate on the overall representation of the module.…”

Section: Introductionmentioning

confidence: 99%

A Modular Hierarchical Model for Paper Quality Evaluation

Deng¹,

Li²,

Yu³

et al. 2023

Artificial Intelligence Advances

View full text Add to dashboard Cite

show abstract

“…Automated grading systems are developed for automatically reviewing research articles, grading programming assignments, poems, short answers, and long answers. Leng, Yu, and Xiong (2019) developed an automated system to review research papers. They used Hierarchical Recurrent Convolutional Neural Network (HRCNN), Multilayer Perceptron (MLP), and Convolutional Neural Network (CNN) models to get semantic, grammatical, and innovative features respectively from the research papers.…”

Section: Related Workmentioning

confidence: 99%

Automatic Grading of Student’s Presentation Skills based on PowerPoint Presentation and Audio

Borade

Kiwelekar

Netak

2022

UPjeng

View full text Add to dashboard Cite

Creating and delivering a presentation for Project, Seminar, and Course work is an important academic activity included in the curricula of undergraduate engineering studies. The presentation should be graded based on the presentation skills, accuracy, and authenticity of the contents covered in the presentation. Educational institutes use rubrics to assess the presentation skills on different grounds, which is a cumbersome task for the teacher when the strength of students is significant. Our main objective is to automatically grade the students' presentation skills in terms of PowerPoint presentations and the student's confidence. The proposed system describes a method and dataset designed to automate grading students' presentation skills. Our research study is divided into two parts. In the first part, the PowerPoint presentation features corresponding to text appearance, tables, charts, images, footer, and hyperlinks are extracted to grade PowerPoint presentations. At the same time, Mel-frequency Cepstral Coefficients, Mel Spectrogram, and Chroma features are extracted from the students' audio to identify confidence in the second part of the study. The audio is recorded at the presentation time. Feature extraction programs are implemented in python using Python-pptx and Librosa library. The tree-based feature selection method is used to remove the irrelevant features. Random Forest Ensemble model gives 100 % accuracy while predicting the grade of PowerPoint presentations. Multilayer Perceptron model gives 88% accuracy while predicting the confidence level of the students. The output of both models is combined to grade the students' presentation skills. The quadratic Weighted Kappa (QWK) score is 0.82, which indicates a significant similarity between automated and human-rated scores.

show abstract

“…In addition to scoring different aspects of a paper, there are also methods proposed to produce an overall score directly. Leng et al (2019) introduced an attention-based framework DeepReviewer that assigns scores for papers on OpenReview based on the semantic, grammatical and innovative features combined. This framework is composed of a hierarchical recurrent convolutional neural network, a customized unsupervised deep context grammar model, an unsupervised high-dimensional spatial density-based innovation model and an attention layer to generate the final review score.…”

Section: Scoringmentioning

confidence: 99%

Automated scholarly paper review: Technologies and challenges

Lin¹,

Song²,

Zhou³

et al. 2021

Preprint

View full text Add to dashboard Cite

Peer review is a widely accepted mechanism for research evaluation, playing a pivotal role in scholarly publishing. However, criticisms have long been leveled on this mechanism, mostly because of its inefficiency and subjectivity. Recent years have seen the application of artificial intelligence (AI) in assisting the peer review process. Nonetheless, with the involvement of humans, such limitations remain inevitable. In this review paper, we propose the concept of automated scholarly paper review (ASPR) and review the relevant literature and technologies to discuss the possibility of achieving a full-scale computerized review process. We further look into the challenges in ASPR with the existing technologies. On the basis of the review and discussion, we conclude that there are already corresponding research and technologies at each stage of ASPR. This verifies that ASPR can be realized in the long term as the relevant technologies continue to develop. The major difficulties in its realization lie in imperfect document parsing and representation, inadequate data, defected human-computer interaction and flawed deep logical reasoning. In the foreseeable future, ASPR and peer review will coexist in a reinforcing manner before ASPR is able to fully undertake the reviewing workload from humans.

show abstract

DeepReviewer: Collaborative Grammar and Innovation Neural Network for Automatic Paper Review

Cited by 12 publications

References 6 publications

A Modular Hierarchical Model for Paper Quality Evaluation

A Modular Hierarchical Model for Paper Quality Evaluation

Automatic Grading of Student’s Presentation Skills based on PowerPoint Presentation and Audio

Automated scholarly paper review: Technologies and challenges

Contact Info

Product

Resources

About