Controlling Text Edition by Changing Answers of Specific Questions

Sha, Lei; Hohenecker, Patrick; Lukasiewicz, Thomas

doi:10.18653/v1/2021.findings-acl.110

Cited by 3 publications

(3 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides the aforementioned paraphrasing and style transfer, prior works have also successfully generated contrastive examples that are useful for model training, evaluation, and explanation. They usually rely on application-specific class labels (Ross et al, 2020;Madaan et al, 2020b;Sha et al, 2021;Akyürek et al, 2020) or heuristic perturbation strategies that needs to be expressed through pairs of original and perturbed sentences (Wu et al, 2021), which are expensive to generalize. Recently, Huang and Chang (2021) designed SynPG, a paraphraser that can mimic parse tree structures learned from non-paired sentences.…”

Section: Related Workmentioning

confidence: 99%

Tailor: Generating and Perturbing Text with Semantic Controls

Ross¹,

Wu²,

Peng³

et al. 2021

Preprint

View full text Add to dashboard Cite

Making controlled perturbations is essential for various tasks (e.g., data augmentation), but building task-specific generators can be expensive. We introduce Tailor, a taskagnostic generation system that perturbs text in a semantically-controlled way. With unlikelihood training, we design Tailor's generator to follow a series of control codes derived from semantic roles. Through modifications of these control codes, Tailor can produce fine-grained perturbations. We implement a set of operations on control codes that can be composed into complex perturbation strategies, and demonstrate their effectiveness in three distinct applications: First, Tailor facilitates the construction of high-quality contrast sets that are lexically diverse, and less biased than original task test data. Second, paired with automated labeling heuristics, Tailor helps improve model generalization through data augmentation: We obtain an average gain of 1.73 on an NLI challenge set by perturbing just ≈5% of training data. Third, without any finetuning overhead, Tailor's perturbations effectively improve compositionality in fine-grained style transfer, outperforming fine-tuned baselines on 6 transfers.

show abstract

Section: Related Workmentioning

confidence: 99%

Tailor: Generating and Perturbing Text with Semantic Controls

Ross¹,

Wu²,

Peng³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Controlled generators have also been successfully used to perturb text for model training, evaluation, and explanation. They usually rely on application-specific labels Madaan et al, 2020b;Sha et al, 2021;Akyürek et al, 2021) or require pairs of original and perturbed sentences (Wu et al, 2021), which are expensive to generalize.…”

Section: Related Workmentioning

confidence: 99%

Tailor: Generating and Perturbing Text with Semantic Controls

Ross¹,

Wu²,

Peng³

et al. 2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Controlled text perturbation is useful for evaluating and improving model generalizability. However, current techniques rely on training a model for every target perturbation, which is expensive and hard to generalize. We present Tailor, a semantically-controlled text generation system. Tailor builds on a pretrained seq2seq model and produces textual outputs conditioned on control codes derived from semantic representations. We craft a set of operations to modify the control codes, which in turn steer generation towards targeted attributes. These operations can be further composed into higher-level ones, allowing for flexible perturbation strategies. We demonstrate the effectiveness of these perturbations in multiple applications. First, we use Tailor to automatically create high-quality contrast sets for four distinct natural language processing (NLP) tasks. These contrast sets contain fewer spurious artifacts and are complementary to manually annotated ones in their lexical diversity. Second, we show that Tailor perturbations can improve model generalization through data augmentation. Perturbing just ∼2% of training data leads to a 5.8-point gain on an NLI challenge set measuring reliance on syntactic heuristics.

show abstract

“…Another concurrent work by Wu et al (2021) presents POLYJUICE, a general-purpose, untargeted counterfactual generator. Very recent work by Sha et al (2021), introduced after the submission of MICE, proposes a method for targeted contrastive editing for Q&A that selects answer-related tokens, masks them, and generates new tokens. Our work differs from these works in our novel framework for efficiently finding minimal edits (MICE Stage 2) and our use of edits as explanations.…”

Section: Counterfactuals Beyond Explanations Concurrent Work Bymentioning

confidence: 99%

Explaining NLP Models via Minimal Contrastive Editing (MiCE)

Ross¹,

Marasovi²,

Peters³

2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

Humans have been shown to give contrastive explanations, which explain why an observed event happened rather than some other counterfactual event (the contrast case).Despite the influential role that contrastivity plays in how humans explain, this property is largely missing from current methods for explaining NLP models. We present MIN-IMAL CONTRASTIVE EDITING (MICE), a method for producing contrastive explanations of model predictions in the form of edits to inputs that change model outputs to the contrast case. Our experiments across three tasks-binary sentiment classification, topic classification, and multiple-choice question answering-show that MICE is able to produce edits that are not only contrastive, but also minimal and fluent, consistent with human contrastive edits. We demonstrate how MICE edits can be used for two use cases in NLP system development-debugging incorrect model outputs and uncovering dataset artifacts-and thereby illustrate that producing contrastive explanations is a promising research direction for model interpretability.

show abstract

Controlling Text Edition by Changing Answers of Specific Questions

Cited by 3 publications

References 29 publications

Tailor: Generating and Perturbing Text with Semantic Controls

Tailor: Generating and Perturbing Text with Semantic Controls

Tailor: Generating and Perturbing Text with Semantic Controls

Explaining NLP Models via Minimal Contrastive Editing (MiCE)

Contact Info

Product

Resources

About