DeepSentiPeer: Harnessing Sentiment in Review Texts to Recommend Peer Review Decisions

Abstract

Automatically validating a research artefact is one of the frontiers in Artificial Intelligence (AI) that directly brings it close to competing with human intellect and intuition. Although criticized sometimes, the existing peer review system still stands as the benchmark of research validation. The present-day peer review process is not straightforward and demands profound domain knowledge, expertise, and intelligence of human reviewer(s), which is somewhat elusive with the current state of AI. However, the peer review texts, which contains rich sentiment information of the reviewer, reflecting his/her overall attitude towards the research in the paper, could be a valuable entity to predict the acceptance or rejection of the manuscript under consideration. Here in this work, we investigate the role of reviewers sentiments embedded within peer review texts to predict the peer review outcome. Our proposed deep neural architecture takes into account three channels of information: the paper, the corresponding reviews, and the review polarity to predict the overall recommendation score as well as the final decision. We achieve significant performance improvement over the baselines (∼ 29% error reduction) proposed in a recently released dataset of peer reviews. An AI of this kind could assist the editors/program chairs as an additional layer of confidence in the final decision making, especially when non-responding/missing reviewers are frequent in present day peer review.

Uncertainty Aware Review Hallucination for Science Article Classification

“…In this short paper, we offer solutions to three particularities of this task that the above approaches do not address: a) Often, the recommendations given by the area chair and the reviewers are in disagreement. Whereas previous studies have used either the former (Kang et al, 2018;Wang and Wan, 2018;Ghosal et al, 2019) or a soft label average of the latter (Stappen et al, 2020) for supervision, we show that both signals comprise complementary information. b) Whereas soft labels de-emphasise subjective articles with disagreeing reviews during training (Stappen et al, 2020), we manage to outperform the latter study by explicitly modelling aleatory uncertainty as an auxiliary prediction task.…”

“…Finally, we have shown in our study that only a representation of the abstract is required as the input both for acceptance and hallucination modelling. Since previous work (Ghosal et al, 2019) has shown that modelling an article based on the entire paper can be beneficial, we also intend to explore the impact of using such a highly expressive article representation for hallucinating review representations.…”

“…Wang and Wan (2018) explored the acceptance task by modelling the abstract via a memory mechanism (Weston et al, 2015), along with one review. Ghosal et al (2019) improved performance on the Peer-Read dataset by utilising sentiment information using the VADER tool (Hutto and Gilbert, 2014) and universal sentence embeddings (Cer et al, 2018). Unfortunately, PeerRead is imbalanced in that the NeurIPS rejected submissions are not included, despite the fact that 90% of the accepted submissions with reviews in PeerRead are from NeurIPS.…”

“…Small available dataset size is a limitation known to the community working on this domain (Kang et al, 2018;Ghosal et al, 2019) -we use the largest database of its kind (Stappen et al, 2020), i. e., the 2 179 preprocessed academic submissions, 5 842 reviews, with corresponding acceptance decisions and reviewer scores from the submission system of Interspeech 2019, shared with us by the technical chairs of the conference. After data cleaning and removal of corrupt entries, the accepted and rejected classes are well-balanced: 50.2 % acceptances, and 49.8 % rejections.…”

“…Natural Language Understanding (NLU) could also offer decision support to the area chair, as argued in (Ghosal et al, 2019;Stappen et al, 2020). Such systems jointly model the entire or part of the article and one (Kang et al, 2018;Wang and Wan, 2018;Ghosal et al, 2019), or a variable number…”