Reformulating Unsupervised Style Transfer as Paraphrase Generation

Abstract

Modern NLP defines the task of style transfer as modifying the style of a given sentence without appreciably changing its semantics, which implies that the outputs of style transfer systems should be paraphrases of their inputs. However, many existing systems purportedly designed for style transfer inherently warp the input's meaning through attribute transfer, which changes semantic properties such as sentiment. In this paper, we reformulate unsupervised style transfer as a paraphrase generation problem, and present a simple methodology based on fine-tuning pretrained language models on automatically generated paraphrase data. Despite its simplicity, our method significantly outperforms state-of-the-art style transfer systems on both human and automatic evaluations. We also survey 23 style transfer papers and discover that existing automatic metrics can be easily gamed and propose fixed variants. Finally, we pivot to a more real-world style transfer setting by collecting a large dataset of 15M sentences in 11 diverse styles, which we use for an in-depth analysis of our system.

Open-Retrieval Conversational Question Answering

“…Diverse Paraphrase Model We now briefly describe the diverse paraphraser [19] used in the training process of the learned weak supervisor. This model is built by fine-tuning GPT2-large [33] using encoder-free seq2seq modeling [46].…”

“…Our learned weak supervisor is based on Transformers [41]. Due to the lack of training data to learn this module, we propose a novel training method for the learned weak supervisor by leveraging a diverse paraphraser [19] to generate the training data. Once the learned weak supervisor is trained, it is frozen and used to facilitate the training of the ORConvQA model.…”

Olá, Bonjour, Salve! XFORMAL: A Benchmark for Multilingual Formality Style Transfer

“…Notably, perplexity tells a different story: all system outputs are significantly better compared to the RULE-BASED systems across configurations and languages. This result denotes that perplexity might not be a reliable metric to measure fluency in this setting, as noticed in Mir et al (2019) and Krishna et al (2020). When it comes to the overall ranking of systems, we observe that the NMT-based ensembles are better than the RULE-BASED baselines for BR-PT and FR, yet by a small margin as denoted by both multi-BLEU and human evaluation.…”

“…Parallel corpora designed for the task at hand are used to train traditional encoder-decoder architectures (Rao and Tetreault, 2018), learn mappings between latent representation of different styles (Shang et al, 2019), or fine-tune pre-trained models (Wang et al, 2019). Other approaches use parallel data from similar tasks to facilitate transfer in the target style via domain adaptation , multi-task learning (Niu et al, 2018;Niu and Carpuat, 2020), and zero-shot transfer (Korotkova et al, 2019) or create pseudo-parallel data via data augmentation techniques (Zhang et al, 2020;Krishna et al, 2020). Approaches that rely on non-parallel data include disentanglement methods based on the idea of learning style-agnostic latent representations (e.g., Shen et al (2017); Hu et al (2017)).…”

StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer

“…Current benchmarks for style transfer focus on high-level style definitions such as transfer of sentiment (Shen et al, 2017;Lample et al, 2019;, politeness (Madaan et al, 2020), formality (Rao and Tetreault, 2018;Krishna et al, 2020), writing styles (Jhamtani et al, 2017;Syed et al, 2020;Jin et al, 2020) and some other styles (Kang and Hovy, 2019). However, these only focus on only high-level styles, unlike STYLEPTB.…”