Why Do Document-Level Polarity Classifiers Fail?

Martins, Karen S.; Melo, Pedro O. S. Vaz de; Santos, Rodrygo L. T.

doi:10.18653/v1/2021.naacl-main.143

Cited by 3 publications

(4 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also note that the human performed worse (Accuracy: 45.83%) on instances where the machine classifier error was higher (Table 12), when compared to the performance (Accuracy: 50%) on instances where the machine classifier error was lower (Table 13), indicating that, in fact, this group of instances may have more misleading texts. Considering the two types of hardest instances proposed by Martins et al [24], for 21 (22%) instances the annotator was not certain about their polarities, that is, 22% of the instances in this sample are neutral hard instances. The accuracy for these instances was only 23% (Table 14), significantly lower than for the instances in which the annotator did not have any doubt: 54.66% (Table 15).…”

Section: Review Score Prediction (Rsp)mentioning

confidence: 81%

“…In the other direction, Krymolowski [19] defines hard instances as those that lead to a higher probability of error. Martins et al [24] proposed a methodology for finding hard instances in movie reviews and showed that such instances are significantly more difficult to classify. To the best of our knowledge, only Wang and Wan [32] evaluated the impact of hard instances on the paper review classification task.…”

Section: Related Workmentioning

confidence: 99%

“…This suggests that the classifier is fairly confident of these mistakes, which may characterize a discrepant instance, i.e., a review text that conveys the opposing polarity of its associated score. Inspired by this conjecture, we conducted the methodology proposed by Martins et al [24] to identify hard instances in polarity classification tasks, which is able to identify texts with discrepant and neutral polarities. To do that, we selected 24 reviews for each of the scenarios that had the highest average probability error, which are: (i) instances of class 𝑆 −− , but predicted as 𝑆 ++ ; (ii) instances of class 𝑆 0− , but predicted as 𝑆 ++ ; (iii) instances of class 𝑆 0+ , but predicted as 𝑆 −− and (iv) instances of class 𝑆 ++ , but predicted as 𝑆 −− .…”

Section: Review Score Prediction (Rsp)mentioning

confidence: 99%

“…A researcher with high peer review experience was asked to read the reviews, to score them (playing the role of the classifier) and indicate if she/he felt in doubt when rating them. According to Martins et al [24], reviews in which the annotator is in doubt about its polarity are neutral hardest instances. On the other hand, reviews for which the annotator is certain about its polarity and makes a mistake are discrepant hardest instances.…”

Section: Review Score Prediction (Rsp)mentioning

confidence: 99%

See 3 more Smart Citations

Between acceptance and rejection

Fernandes

Melo

2022

Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries

View full text Add to dashboard Cite

The peer review process is the main academic resource to ensure that science advances and is disseminated. To contribute to this important process, classification models capable of predicting the score of a review text (RSP) and the final decision of a paper (PDP) were created. But what challenges prevent us from having a fully efficient system responsible for these tasks? And how far are we from having an automated system to take care of these two tasks? To answer these questions, in this work, we evaluated the general performance of existing state-of-the-art models for RSP and PDP tasks, and investigated what types of instances these models tend to have difficulty classifying and how impactful they are. We found, for example, that the performance of a model to predict the final decision of a paper is 23.31% lower when it is exposed to difficult instances and that the classifiers make mistake with a very high confidence. These and other results lead us to conclude that there are groups of instances that can negatively impact the model's performance. That way, the current state-of-the-art models have potential to helping editors to decide whether to approve or reject a paper, however we are still far from having a system that is fully responsible for scoring a paper and decide if it will be accepted or rejected. CCS CONCEPTS• Computing methodologies → Natural language processing; Supervised learning by classification.

show abstract

Section: Review Score Prediction (Rsp)mentioning

confidence: 81%

Section: Related Workmentioning

confidence: 99%

Section: Review Score Prediction (Rsp)mentioning

confidence: 99%

Section: Review Score Prediction (Rsp)mentioning

confidence: 99%

See 2 more Smart Citations

Between acceptance and rejection

Fernandes

Melo

2022

Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries

View full text Add to dashboard Cite

show abstract

Enhancing the examination of obstacles in an automated peer review system

Fernandes,

Vaz-de-Melo

2023

Int J Digit Libr

View full text Add to dashboard Cite

A Comparative Survey of Instance Selection Methods applied to Non-Neural and Transformer-Based Text Classification

et al. 2023

View full text Add to dashboard Cite

Progress in Natural Language Processing (NLP) has been dictated by the rule of more : more data, more computing power, more complexity, best exemplified by Deep Learning Transformers. However, training (or fine-tuning) large dense models for specific applications usually requires significant amounts of computing resources. One way to ameliorate this problem is through data engineering (DE) instead of the algorithmic or hardware perspectives. Our focus here is an under-investigated DE technique, with enormous potential in the current scenario – Instance Selection (IS) (aka, Selective Sampling; Prototype Selection). The IS goal is to reduce the training set size by removing noisy or redundant instances while maintaining or improving the effectiveness (accuracy) of the trained models and reducing the training process cost . We survey classical and recent state-of-the-art IS techniques and provide a scientifically sound comparison of IS methods applied to an essential NLP task - Automatic Text Classification (ATC). IS methods have been normally applied to small tabular datasets and have not been systematically compared in ATC. We consider several neural and non-neural state-of-the-art ATC solutions and many datasets. We answer several research questions based on tradeoffs induced by a tripod (Effectiveness, Efficiency, Training-Set Reduction). Our answers reveal an enormous unfulfilled potential for IS solutions. Specially, we show that in 12 out of 19 datasets, specific IS methods - namely Condensed Nearest Neighbor (CNN), Local Set-based Smoother (LSSm) and Local Set Border Selector (LSBo) – can reduce the size of the training set without effectiveness losses. Furthermore, in the case of fine-tuning the transformer methods, the IS methods reduce the amount of data needed, without losing effectiveness and with considerable training-time gains.

show abstract

Why Do Document-Level Polarity Classifiers Fail?

Cited by 3 publications

References 27 publications

Between acceptance and rejection

Between acceptance and rejection

Enhancing the examination of obstacles in an automated peer review system

A Comparative Survey of Instance Selection Methods applied to Non-Neural and Transformer-Based Text Classification

Contact Info

Product

Resources

About