Breaking Sentiment Analysis of Movie Reviews

Staliūnaitė, Ieva; Bonfil, B.

doi:10.18653/v1/w17-5410

Cited by 9 publications

(7 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using recast NLI, Poliak et al (2018a) probe for semantic phenomena in neural machine translation encoders. Staliūnaite and Bonfil (2017); Mahler et al (2017);Ribeiro et al (2018) use similar strategies to our structural mutation method, although their primary goal was to break existing systems by adversarial modifications rather than to compare different models. Ribeiro et al (2018) and our work both test for proper comprehension of the modified expressions, but our modifications are designed to induce semantic changes whereas their modifications are intended to preserve the original meaning.…”

Section: Related Workmentioning

confidence: 99%

Probing What Different NLP Tasks Teach Machines about Function Word Comprehension

Kim¹,

Patel²,

Poliak³

et al. 2019

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)

View full text Add to dashboard Cite

We introduce a set of nine challenge tasks that test for the understanding of function words. These tasks are created by structurally mutating sentences from existing datasets to target the comprehension of specific types of function words (e.g., prepositions, wh-words). Using these probing tasks, we explore the effects of various pretraining objectives for sentence encoders (e.g., language modeling, CCG supertagging and natural language inference (NLI)) on the learned representations. Our results show that pretraining on language modeling performs the best on average across our probing tasks, supporting its widespread use for pretraining state-of-the-art NLP models, and CCG supertagging and NLI pretraining perform comparably. Overall, no pretraining objective dominates across the board, and our function word probing tasks highlight several intuitive differences between pretraining objectives, e.g., that NLI helps the comprehension of negation.

show abstract

Section: Related Workmentioning

confidence: 99%

Probing What Different NLP Tasks Teach Machines about Function Word Comprehension

Kim¹,

Patel²,

Poliak³

et al. 2019

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)

View full text Add to dashboard Cite

show abstract

“…Manual construction informed by expert knowledge: Instances that exploit world knowledge, semantics, pragmatics, morphology and syntactic variation have been written and compiled into challenge datasets for Machine Translation (Isabelle et al, 2017), Sentiment Analysis (Mahler et al, 2017;Staliūnaite and Bonfil, 2017) and Natural Language Understanding (Levesque, 2013). While these instances are expensive to construct, the attacker would have a high degree of confidence that the instances are correct and therefore correctness is not incorporated into the scoring metrics of any of these works.…”

Section: Methods For Adversarial Evaluationmentioning

confidence: 99%

Evaluating adversarial attacks against multiple fact verification systems

Thorne¹,

Vlachos²,

Christodoulopoulos³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

Automated fact verification has been progressing owing to advancements in modeling and availability of large datasets. Due to the nature of the task, it is critical to understand the vulnerabilities of these systems against adversarial instances designed to make them predict incorrectly. We introduce two novel scoring metrics, attack potency and system resilience which take into account the correctness of the adversarial instances, an aspect often ignored in adversarial evaluations. We consider six fact verification systems from the recent Fact Extraction and VERification (FEVER) challenge: the four best-scoring ones and two baselines. We evaluate adversarial instances generated by a recently proposed state-of-theart method, a paraphrasing method, and rulebased attacks devised for fact verification. We find that our rule-based attacks have higher potency, and that while the rankings among the top systems changed, they exhibited higher resilience than the baselines.

show abstract

“…Manual Construction Small adversarial datasets have been manually constructed and successfully used to identify limitations in Machine Translation [3,11], Sentiment Analysis [15,25] and Natural Language Understanding [14] systems. Instances are generated that exploit world knowledge, semantics, pragmatics, morphology and syntactic variations.…”

Section: Related Workmentioning

confidence: 99%

Adversarial attacks against Fact Extraction and VERification

Thorne,

Vlachos

2019

Preprint

View full text Add to dashboard Cite

This paper describes a baseline for the second iteration of the Fact Extraction and VERification shared task (FEVER2.0) which explores the resilience of systems through adversarial evaluation. We present a collection of simple adversarial attacks against systems that participated in the first FEVER shared task. FEVER modeled the assessment of truthfulness of written claims as a joint information retrieval and natural language inference task using evidence from Wikipedia. A large number of participants made use of deep neural networks in their submissions to the shared task. The extent as to whether such models understand language has been the subject of a number of recent investigations and discussion in literature. In this paper, we present a simple method of generating entailmentpreserving and entailment-altering perturbations of instances by common patterns within the training data. We find that a number of systems are greatly affected with absolute losses in classification accuracy of up to 29% on the newly perturbed instances. Using these newly generated instances, we construct a sample submission for the FEVER2.0 shared task. Addressing these types of attacks will aid in building more robust fact-checking models, as well as suggest directions to expand the datasets.

show abstract

Breaking Sentiment Analysis of Movie Reviews

Cited by 9 publications

References 3 publications

Probing What Different NLP Tasks Teach Machines about Function Word Comprehension

Probing What Different NLP Tasks Teach Machines about Function Word Comprehension

Evaluating adversarial attacks against multiple fact verification systems

Adversarial attacks against Fact Extraction and VERification

Contact Info

Product

Resources

About