Breaking NLI Systems with Sentences that Require Simple Lexical Inferences

Glockner, Max; Shwartz, Vered; Goldberg, Yoav

doi:10.48550/arxiv.1805.02266

Cited by 9 publications

(17 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, many of such efforts have focused on only a specific task, such as sentiment analysis, question answering, etc. and left unaddressed the challenge of defending NLP models against a generic adversary optimizing in the input language for multiple tasks [40], [41], [48], [49], [102]. Moreover, most research works focus on how to develop certain types of concrete adversarial examples in a constrained adversarial setting.…”

Section: E Insights and Open Directionsmentioning

confidence: 99%

“…The classification accuracy has been utilized by numerous research works [34], [35], [40], [41], [45], [59], [103], [105], [106]. For example, in [59], Zhang et al used the classifi-cation accuracy metric to evaluate their proposed Metropolis-Hastings Sampling Algorithm (MHA) and demonstrated that MHA under classification accuracy outperforms the baseline model on attacking capability.…”

Section: Classification Accuracymentioning

confidence: 99%

See 1 more Smart Citation

Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions

Omar¹,

Choi²,

Nyang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent natural language processing (NLP) techniques have accomplished high performance on benchmark datasets, primarily due to the significant improvement in the performance of deep learning. The advances in the research community have led to great enhancements in state-of-the-art production systems for NLP tasks, such as virtual assistants, speech recognition, and sentiment analysis. However, such NLP systems still often fail when tested with adversarial attacks. The initial lack of robustness exposed troubling gaps in current models' language understanding capabilities, creating problems when NLP systems are deployed in real life. In this paper, we present a structured overview of NLP robustness research by summarizing the literature in a systemic way across various dimensions. We then take a deep-dive into the various dimensions of robustness, across techniques, metrics, embeddings, and benchmarks. Finally, we argue that robustness should be multi-dimensional, provide insights into current research, identify gaps in the literature to suggest directions worth pursuing to address these gaps.

show abstract

Section: E Insights and Open Directionsmentioning

confidence: 99%

Section: Classification Accuracymentioning

confidence: 99%

Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions

Omar¹,

Choi²,

Nyang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…(Wang et al, 2019) introduces a swapping evaluation method, which means changing the distribution of words by swapping a premise with its corresponding hypothesis to test the robustness of models. Also, new test datasets are proposed, e.g., Glockner test set (Glockner et al, 2018). In the Glockner test dataset, premises are taken from the SNLI training set, and hypotheses are generated by replacing a single word in its corresponding premise sentence.…”

Section: Evaluation Of Models For Nlimentioning

confidence: 99%

“…We evaluate the proposed models on the widely used NLI datasets, i.e., SNLI and MultiNLI (MultiNLI-match and MultiNLI-mismatch). We also test our models with the Glockner testset (Glockner et al, 2018). These datasets share the same target of a 3-way prediction: determining the relation in a premise-hypothesis pair to be either entailment, neutral, or contradiction.…”

Section: Datamentioning

confidence: 99%

“…We evaluate the proposed models on the widely used NLI benchmarks, including the Stanford Natural Language Inference (SNLI) corpus (Bowman et al, 2015), Multi-Genre Natural Language Inference (MultiNLI) corpus (Williams et al, 2017), and a newly introduced diagnostic Glockner dataset (Glockner et al, 2018). Our results show that the proposed models perform better on these datasets than previous BERT-based state-of-the-art Unsupervised pre-trained models (Peters et al, 2018;Radford et al, 2018;Devlin et al, 2018;Radford et al, 2019;Song et al, 2019) have proven to be effective in achieving great improvements in many NLP tasks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Enhancing Unsupervised Pretraining with External Knowledge for Natural Language Inference

Yang

Zhu

Zhao

et al. 2019

Advances in Artificial Intelligence

View full text Add to dashboard Cite

While recent research on natural language inference has considerably benefited from large annotated datasets (Williams et al., 2017;Bowman et al., 2015), the amount of inferencerelated knowledge (including commonsense) provided in the annotated data is still rather limited. There have been two lines of approaches that can be used to further address the limitation: (1) unsupervised pretraining can leverage knowledge in much larger unstructured text data; (2) structured (often humancurated) knowledge has started to be considered in neural-network-based models for NLI. An immediate question is whether these two approaches complement each other, or how to develop models that can bring together their advantages. In this paper, we propose models that leverage structured knowledge in different components of pre-trained models. Our results show that the proposed models perform better than previous BERT-based state-of-the-art models. Although our models are proposed for NLI, they can be easily extended to other sentence or sentence-pair classification problems.

show abstract

Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions

et al. 2022

View full text Add to dashboard Cite

Recent natural language processing (NLP) techniques have accomplished high performance on benchmark data sets, primarily due to the significant improvement in the performance of deep learning. The advances in the research community have led to great enhancements in state-of-the-art production systems for NLP tasks, such as virtual assistants, speech recognition, and sentiment analysis. However, such NLP systems still often fail when tested with adversarial attacks. The initial lack of robustness exposed troubling gaps in current models' language understanding capabilities, creating problems when NLP systems are deployed in real life. In this paper, we present a structured overview of NLP robustness research by summarizing the literature in a systemic way across various dimensions. We then take a deepdive into the various dimensions of robustness, across techniques, metrics, embedding, and benchmarks. Finally, we argue that robustness should be multi-dimensional, provide insights into current research, identify gaps in the literature to suggest directions worth pursuing to address these gaps INDEX TERMS Natural Language Processing; Adversarial Attacks; Robustness.

show abstract

Breaking NLI Systems with Sentences that Require Simple Lexical Inferences

Cited by 9 publications

References 0 publications

Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions

Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions

Enhancing Unsupervised Pretraining with External Knowledge for Natural Language Inference

Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions

Contact Info

Product

Resources

About