An Analysis of Natural Language Inference Benchmarks through the Lens of Negation

Hossain, Mosharaf; Kovatchev, Venelin; Dutta, Pranoy; Kao, Tiffany; Wei, Elizabeth; Blanco, Eduardo

doi:10.18653/v1/2020.emnlp-main.732

Cited by 27 publications

(23 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results of "asking" entailment models whether the text is a suggestion or not are shown in Table 4. The difference in performance of all models on test sets of Subtasks A and B can be explained by the differences in target classes distributions; we also note that Transformer-based models often struggle with negations [9]. Similar to [35], using definitions seemingly does not guarantee better results for entailment-based zero-shot text (suggestion) classification.…”

Section: Resultsmentioning

confidence: 92%

“…Further, we have selected a list of candidates from those hyponyms to be later mapped to suggestions: direction.n.06, guidance.n.01, offer.n.02, promotion.n.01, proposal.n.01, reminder.n.01, request.n.01, submission.n.01. We have formulated the labels as "This text is a [LEMMA]", where [LEMMA] is the first lemma in the WordNet lemma synset list 9 . We have used the development set of the SemEval2019 Task 9, Subtask A to find the best subset of candidate labels in terms of F1-measure of the "suggestion" class.…”

Section: Approachmentioning

confidence: 99%

See 1 more Smart Citation

Near-Zero-Shot Suggestion Mining with a Little Help from WordNet

Alekseev¹,

Tutubalina²,

Kwon³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this work we explore the constructive side of online reviews: advice, tips, requests, and suggestions that users provide about goods, venues, services and other items of interest. To reduce training costs and annotation efforts needed to build a classifier for a specific label set, we present and evaluate several entailment-based zero-shot approaches to suggestion classification in a label-fully-unseen fashion. In particular, we introduce the strategy of assigning target class labels to sentences in English language with user intentions, which significantly improves prediction quality. The proposed strategies are evaluated with a comprehensive experimental study that validated our results both quantitatively and qualitatively.

show abstract

Section: Resultsmentioning

confidence: 92%

Section: Approachmentioning

confidence: 99%

Near-Zero-Shot Suggestion Mining with a Little Help from WordNet

Alekseev¹,

Tutubalina²,

Kwon³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Pretrained language models fail to make correct predictions in the presence of negation or even to distinguish between positive and negative sentences (Ettinger, 2020;Kassner and Schütze, 2020). Hossain et al (2020) finetune several different pre-trained language models for natural language inference and show that performance on instances containing negation deteriorates.…”

Section: Negationmentioning

confidence: 99%

Investigating Negation in Pre-trained Vision-and-language Models

Dobreva¹,

Keller²

2021

Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

View full text Add to dashboard Cite

Pre-trained vision-and-language models have achieved impressive results on a variety of tasks, including ones that require complex reasoning beyond object recognition. However, little is known about how they achieve these results or what their limitations are. In this paper, we focus on a particular linguistic capability, namely the understanding of negation. We borrow techniques from the analysis of language models to investigate the ability of pretrained vision-and-language models to handle negation. We find that these models severely underperform in the presence of negation.

show abstract

“…As a preliminary step in the data augmentation process, our in-domain experts rewrote existing responses to improve the balance of the corpus. We used strategy similar to the one used in Hossain et al (2020). We ran statistical and machine learning experiments to ensure that the additional examples do not introduce biases.…”

Section: Data Augmentationmentioning

confidence: 99%

Can vectors read minds better than experts? Comparing data augmentation strategies for the automated scoring of children’s mindreading ability

Kovatchev¹,

Smith²,

Lee³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

Self Cite

View full text Add to dashboard Cite

In this paper we implement and compare 7 different data augmentation strategies for the task of automatic scoring of children's ability to understand others' thoughts, feelings, and desires (or "mindreading").We recruit in-domain experts to re-annotate augmented samples and determine to what extent each strategy preserves the original rating. We also carry out multiple experiments to measure how much each augmentation strategy improves the performance of automatic scoring systems. To determine the capabilities of automatic systems to generalize to unseen data, we create UK-MIND-20 -a new corpus of children's performance on tests of mindreading, consisting of 10,320 question-answer pairs.We obtain a new state-of-the-art performance on the MIND-CA corpus, improving macro-F1-score by 6 points. Results indicate that both the number of training examples and the quality of the augmentation strategies affect the performance of the systems. The taskspecific augmentations generally outperform task-agnostic augmentations. Automatic augmentations based on vectors (GloVe, FastText) perform the worst.We find that systems trained on MIND-CA generalize well to UK-MIND-20. We demonstrate that data augmentation strategies also improve the performance on unseen data.

show abstract

An Analysis of Natural Language Inference Benchmarks through the Lens of Negation

Cited by 27 publications

References 37 publications

Near-Zero-Shot Suggestion Mining with a Little Help from WordNet

Near-Zero-Shot Suggestion Mining with a Little Help from WordNet

Investigating Negation in Pre-trained Vision-and-language Models

Can vectors read minds better than experts? Comparing data augmentation strategies for the automated scoring of children’s mindreading ability

Contact Info

Product

Resources

About