A Probabilistic Formulation of Unsupervised Text Style Transfer

Abstract

We present a deep generative model for unsupervised text style transfer that unifies previously proposed non-generative techniques. Our probabilistic approach models non-parallel data from two domains as a partially observed parallel corpus. By hypothesizing a parallel latent sequence that generates each observed sequence, our model learns to transform sequences from one domain to another in a completely unsupervised fashion. In contrast with traditional generative sequence models (e.g. the HMM), our model makes few assumptions about the data it generates: it uses a recurrent language model as a prior and an encoder-decoder as a transduction distribution. While computation of marginal data likelihood is intractable in this model class, we show that amortized variational inference admits a practical surrogate. Further, by drawing connections between our variational objective and other recent unsupervised style transfer and machine translation techniques, we show how our probabilistic view can unify some known non-generative objectives such as backtranslation and adversarial loss. Finally, we demonstrate the effectiveness of our method on a wide range of unsupervised style transfer tasks, including sentiment transfer, formality transfer, word decipherment, author imitation, and related language translation. Across all style transfer tasks, our approach yields substantial gains over state-of-the-art non-generative baselines, including the state-of-the-art unsupervised machine translation techniques that our approach generalizes. Further, we conduct experiments on a standard unsupervised machine translation task and find that our unified approach matches the current state-of-the-art. 1 * Equal Contribution. 1 Code and data are available at https://github.com/cindyxinyiwang/deep-latent-sequence-model. 2 Notably, some tasks we evaluate on do change content to some degree, such as sentiment transfer, but for conciseness we use the term "style transfer" nonetheless.

Style Pooling: Automatic Text Style Obfuscation for Improved Classification Fairness

“…Training is accomplished using an approach from (He et al, 2020): We employ seq2seq inference networks and use an amortized inference scheme similar to that used in a conventional VAE, but for sequential discrete latents. Ideally, learning should directly optimize the log data likelihood, which is the marginal shown in Eq.…”

“…Fundamentally, our framework is concerned with stylistic features of human-generated text. Thus, a large body of prior work on methods for unsupervised style transfer are related to our approach (Santos et al, 2018;Yang et al, 2018;Luo et al, 2019;He et al, 2020). There is also a vast body of work on style obfuscation (Emmery et al, 2018;Reddy and Knight, 2016;Bevendorff et al, 2019;Shetty et al, 2018).…”

Multi-Pair Text Style Transfer for Unbalanced Data via Task-Adaptive Meta-Learning

“…We now discuss how to find the most suitable value of each balancing variable. We employ the variational inference framework from probabilistic MAML (He et al, 2020) and TAML to extract the task-specific information. The variational inference framework is used to compute posterior distributions for the balancing variables z τ , γ τ , ω τ .…”

“…If parallel training data is available, a wide range of supervised techniques in machine translation (e.g., Seq2Seq models (Bahdanau et al, 2014) and Transformers (Vaswani et al, 2017)) can also be applied to style transfer problems. For non-parallel data, He et al (2020) proposed a probabilistic formulation that models non-parallel data from two domains as a partially observed parallel corpus, and learn the style transfer model in a completely unsupervised fashion. Unsupervised machine translation method has also been adapted to this setting (Zhang et al, 2018).…”

An Empirical Study of Extrapolation in Text Generation with Scalar Control

“…Recent work has explored a variety text generation tasks that condition on a control variable to specify a desired trait of the output. Examples include summarization conditioned on a desired output length (Fan et al, 2018), paraphrase generation conditioned on a parse tree (Krishna et al, 2020), style transfer conditioned on sentiment (He et al, 2020b), and more (Hu et al, 2017;Li et al, 2018;Fu et al, 2018;He et al, 2020a). In this work, we specifically focus on text generation tasks that condition on a scalar control variable, as depicted in Figure 1.…”