Politeness Transfer: A Tag and Generate Approach

Madaan, Aman; Parekh, Tanmay; Póczos, Barnabás; Neubig, Graham; Yang, Yiming; Salakhutdinov, Ruslan; Black, Alan W.; Prabhumoye, Shrimai

doi:10.48550/arxiv.2004.14257

Cited by 16 publications

(31 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the first set of experiments, we rely on four models: Support Vector Machines (SVMs), Bidirectional Long Short-Term Memory Networks (BiLSTMs) with Self-Attention (Graves & Schmidhuber, 2005), BERT (Devlin et al, 2019), and Longformer (Beltagy et al, 2020). For the second set of experiments, we rely on four state of the art style transfer models representative of different methodologies, each representative of a different approach to automatically generate new examples with flipped labels (Hu et al, 2017;Li et al, 2018;Sudhakar et al, 2019;Madaan et al, 2020). To evaluate classifier performance on the resulting augmented data, we make use of SVM, Naive Bayes (NB), BiLSTM w/ SA, and BERT.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Explaining The Efficacy of Counterfactually Augmented Data

Kaushik¹,

Hovy²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

In attempts to produce machine learning models less reliant on spurious patterns in training data, researchers have recently proposed a human-in-the-loop process for generating counterfactually augmented datasets. As applied in NLP, given some documents and their (initial) labels, humans are tasked with revising the text to make a (given) counterfactual label applicable. Importantly, the instructions prohibit edits that are not necessary to flip the applicable label. Models trained on the augmented (original and revised) data have been shown to rely less on semantically irrelevant words and to generalize better out-of-domain. While this work draws on causal thinking, casting edits as interventions and relying on human understanding to assess outcomes, the underlying causal model is not clear nor are the principles underlying the observed improvements in out-of-domain evaluation. In this paper, we explore a toy analog, using linear Gaussian models. Our analysis reveals interesting relationships between causal models, measurement noise, out-of-domain generalization, and reliance on spurious signals.Interestingly our analysis suggests that data corrupted by adding noise to causal features will degrade out-of-domain performance, while noise added to non-causal features may make models more robust out-of-domain. This analysis yields interesting insights that help to explain the efficacy of counterfactually augmented data. Finally, we present a large-scale empirical study that supports this hypothesis.

show abstract

Section: Resultsmentioning

confidence: 99%

“…For Hu et al (2017),4 Sudhakar et al (2019),5 andMadaan et al (2020),6 we found the default hyperparameters used by the authors to work best on our task. In case ofLi et al (2018), 7 we followed the training schedule presented in the paper.…”

mentioning

confidence: 85%

Explaining The Efficacy of Counterfactually Augmented Data

Kaushik¹,

Hovy²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…What is a style in natural languages? Existing style transfer benchmarks primarily focus on individual high-level stylistic changes across sentiment (Shen et al, 2017), formality (Rao and Tetreault, 2018), politeness (Madaan et al, 2020), and writing styles (Jhamtani et al, 2017). Figure 1 provides some motivating examples to show that the high-level style transfers as commonly studied in existing benchmarks (e.g.…”

Section: Fine-grained Compositional Style Transfermentioning

confidence: 99%

“…Current benchmarks for style transfer focus on high-level style definitions such as transfer of sentiment (Shen et al, 2017;Lample et al, 2019;, politeness (Madaan et al, 2020), formality (Rao and Tetreault, 2018;Krishna et al, 2020), writing styles (Jhamtani et al, 2017;Syed et al, 2020;Jin et al, 2020) and some other styles (Kang and Hovy, 2019). However, these only focus on only high-level styles, unlike STYLEPTB.…”

Section: Related Workmentioning

confidence: 99%

“…Computational models for style transfer span statistical NLP methods (Hovy, 1987;Xu et al, 2012), neural generative models (Prabhumoye et al, 2018;Lample et al, 2019;He et al, 2020), and Retrieve-and-Edit approaches Sudhakar et al, 2019;Madaan et al, 2020). These approaches work for a predefined set of styles but are unable to generalize to compositions of styles.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer

Lyu¹,

Liang²,

Pham³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Text style transfer aims to controllably generate text with targeted stylistic changes while maintaining core meaning from the source sentence constant. Many of the existing style transfer benchmarks primarily focus on individual high-level semantic changes (e.g. positive to negative), which enable controllability at a high level but do not offer fine-grained control involving sentence structure, emphasis, and content of the sentence. In this paper, we introduce a large-scale benchmark, STYLEPTB, with (1) paired sentences undergoing 21 fine-grained stylistic changes spanning atomic lexical, syntactic, semantic, and thematic transfers of text, as well as (2) compositions of multiple transfers which allow modeling of fine-grained stylistic changes as building blocks for more complex, high-level transfers. By benchmarking existing methods on STYLEPTB, we find that they struggle to model fine-grained changes and have an even more difficult time composing multiple styles. As a result, STYLEPTB brings novel challenges that we hope will encourage future research in controllable text style transfer, compositional models, and learning disentangled representations. Solving these challenges would present important steps towards controllable text generation.

show abstract