Behavior Analysis of NLI Models: Uncovering the Influence of Three Factors on Robustness

Abstract

Natural Language Inference is a challenging task that has received substantial attention, and state-of-the-art models now achieve impressive test set performance in the form of accuracy scores. Here, we go beyond this single evaluation metric to examine robustness to semantically-valid alterations to the input data. We identify three factors -insensitivity, polarity and unseen pairs -and compare their impact on three SNLI models under a variety of conditions. Our results demonstrate a number of strengths and weaknesses in the models' ability to generalise to new in-domain instances. In particular, while strong performance is possible on unseen hypernyms, unseen antonyms are more challenging for all the models. More generally, the models suffer from an insensitivity to certain small but semantically significant alterations, and are also often influenced by simple statistical correlations between words and training labels. Overall, we show that evaluations of NLI models can benefit from studying the influence of factors intrinsic to the models or found in the dataset used.

Analysis Methods in Neural Language Processing: A Survey

“…Several datasets were constructed by modifying or extracting examples from existing datasets. For instance, Sanchez et al (2018) and Glockner et al (2018) extracted examples from SNLI (Bowman et al, 2015) and replaced specific words such as hypernyms, synonyms, and antonyms, followed by manual verification. Linzen et al (2016), on the other hand, extracted examples of subject-verb agreement from raw texts using heuristics, resulting in a large-scale dataset.…”

Comparing Attention-Based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehension

“…They showed with adversarial examples that most models can be easily tricked by modifications on the data which do not confuse humans. Similarly, Sanchez et al (2018) performed controlled experiments on the robustness of several Natural Language Inference models by altering hypernym, hyponym, and antonym relations in the data. Both studies revealed a major weakness of the models: They largely rely on pattern matching instead of human decision-making processes as required in the tasks, including heuristics (Gigerenzer and Gaissmaier, 2011) and elimination by aspects (Tversky, 1972).…”

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations

“…Snorkel (Ratner et al, 2017), Hard/easy sets (Gururangan et al, 2018) Errudite Compositional-sensitivity Transformations NLPAug (Ma, 2019) Counterfactuals (Kaushik et al, 2019), Stress test (Naik et al, 2018), Bias factors (Sanchez et al, 2018) (Cooper et al, 1994), RTE (Dagan et al, 2005), SICK (Marelli et al, 2014), SNLI , MNLI (Williams et al, 2018), Checklist (Ribeiro et al, 2020) HANS (McCoy et al, 2019b), Quantified NLI (Geiger et al, 2018), MPE (Lai et al, 2017), EQUATE (Ravichander et al, 2019), DNC , ImpPres (Jeretic et al, 2020), Systematicity (Yanaka et al, 2020) ConjNLI (Saha et al, 2020), SherLIiC (Schmitt and Schütze, 2019) Example: author new movie reviews in the style of a newspaper columnist.…”