Breaking NLI Systems with Sentences that Require Simple Lexical Inferences

Glockner, Max; Shwartz, Vered; Goldberg, Yoav

doi:10.18653/v1/p18-2103

Cited by 301 publications

(328 citation statements)

References 19 publications

Supporting

Mentioning

318

Contrasting

Order By: Relevance

“…This contrasts with past work on adversarial examples (e.g. Jia and Liang, 2017;Glockner et al, 2018;Belinkov and Bisk, 2018) which consider cases where an out-of-distribution test set is constructed to be adversarial.…”

Section: Introductionmentioning

confidence: 78%

HellaSwag: Can a Machine Really Finish Your Sentence?

Zellers¹,

Holtzman²,

Bisk³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

421

348

View full text Add to dashboard Cite

Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as "A woman sits at a piano," a machine must select the most likely followup: "She sets her fingers on the keys." With the introduction of BERT (Devlin et al., 2018), near human-level performance was reached. Does this mean that machines can perform human level commonsense inference?In this paper, we show that commonsense inference still proves difficult for even stateof-the-art models, by presenting HellaSwag, a new challenge dataset. Though its questions are trivial for humans (ą95% accuracy), state-of-the-art models struggle (ă48%). We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers. AF proves to be surprisingly robust. The key insight is to scale up the length and complexity of the dataset examples towards a critical 'Goldilocks' zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models.Our construction of HellaSwag, and its resulting difficulty, sheds light on the inner workings of deep pretrained models. More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges. 1 A New York Times article at https://nyti.ms/2DycutY. . 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In arXiv preprint arXiv:1506.06724.

show abstract

Section: Introductionmentioning

confidence: 78%

HellaSwag: Can a Machine Really Finish Your Sentence?

Zellers¹,

Holtzman²,

Bisk³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

421

348

View full text Add to dashboard Cite

show abstract

“…These datasets contain downward inferences, but they are designed not to require lexical knowledge. There are also NLI datasets which expand lexical knowledge by replacing words using lexical rules (Monz and de Rijke, 2001;Glockner et al, 2018;Naik et al, 2018;Poliak et al, 2018a). In these works, however, little attention has been paid to downward inferences.…”

Section: Upwardmentioning

confidence: 99%

HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning

Yanaka

Mineshima

Bekki

et al. 2019

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)

View full text Add to dashboard Cite

Large crowdsourced datasets are widely used for training and evaluating neural models on natural language inference (NLI). Despite these efforts, neural models have a hard time capturing logical inferences, including those licensed by phrase replacements, socalled monotonicity reasoning. Since no large dataset has been developed for monotonicity reasoning, it is still unclear whether the main obstacle is the size of datasets or the model architectures themselves. To investigate this issue, we introduce a new dataset, called HELP, for handling entailments with lexical and logical phenomena. We add it to training data for the state-of-the-art neural models and evaluate them on test sets for monotonicity phenomena. The results showed that our data augmentation improved the overall accuracy. We also find that the improvement is better on monotonicity inferences with lexical replacements than on downward inferences with disjunction and modification. This suggests that some types of inferences can be improved by our data augmentation while others are immune to it.

show abstract

“…A. Glockner et al (2018) This dataset is created by modifying SNLI examples with single word replacements of different lexical relations, based on WordNet. It tests lexical inferences and relatively simple world knowledge.…”

Section: Overall Results and Analysismentioning

confidence: 99%

Adversarial Analysis of Natural Language Inference Systems

Chien

Kalita

2020

2020 IEEE 14th International Conference on Semantic Computing (ICSC)

View full text Add to dashboard Cite

The release of large natural language inference (NLI) datasets like SNLI and MNLI have led to rapid development and improvement of completely neural systems for the task. Most recently, heavily pre-trained, Transformer-based models like BERT and MT-DNN have reached near-human performance on these datasets. However, these standard datasets have been shown to contain many annotation artifacts, allowing models to shortcut understanding using simple fallible heuristics, and still perform well on the test set. So it is no surprise that many adversarial (challenge) datasets have been created that cause models trained on standard datasets to fail dramatically. Although extra training on this data generally improves model performance on just that type of data, transferring that learning to unseen examples is still partial at best. This work evaluates the failures of state-of-the-art models on existing adversarial datasets that test different linguistic phenomena, and find that even though the models perform similarly on MNLI, they differ greatly in their robustness to these attacks. In particular, we find syntax-related attacks to be particularly effective across all models, so we provide a fine-grained analysis and comparison of model performance on those examples. We draw conclusions about the value of model size and multi-task learning (beyond comparing their standard test set performance), and provide suggestions for more effective training data.

show abstract

Breaking NLI Systems with Sentences that Require Simple Lexical Inferences

Cited by 301 publications

References 19 publications

HellaSwag: Can a Machine Really Finish Your Sentence?

HellaSwag: Can a Machine Really Finish Your Sentence?

HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning

Adversarial Analysis of Natural Language Inference Systems

Contact Info

Product

Resources

About