Disentangled Representation Learning for Non-Parallel Text Style Transfer

Abstract

This paper tackles the problem of disentangling the latent variables of style and content in language models. We propose a simple yet effective approach, which incorporates auxiliary multi-task and adversarial objectives, for label prediction and bag-of-words prediction, respectively. We show, both qualitatively and quantitatively, that the style and content are indeed disentangled in the latent space. This disentangled latent representation learning method is applied to style transfer on non-parallel corpora. We achieve substantially better results in terms of transfer accuracy, content preservation and language fluency, in comparison to previous state-of-the-art approaches. 1

Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation

“…Hu et al (2017) propose a new neural generative model which combines variational auto-encoders and holistic attribute discriminators for the effective imposition of semantic structures. Following their work, many methods (Fu et al, 2018;John et al, 2018;Zhang et al, 2018a,b) has been proposed based on standard encoder-decoder architecture.…”

Generating Sentences from Disentangled Syntactic and Semantic Spaces

“…where p(w|z syn ) is the predicted distribution. BoW has been explored by previous work (Weng et al, 2017;John et al, 2018), showing good ability of preserving semantics. For the syntactic space, the multi-task loss trains a model to predict syntax on z syn .…”

“…In particular, we introduce two continuous latent variables to capture semantics and syntax, respectively. To separate the semantic and syntactic information from each other, we borrow the adversarial approaches from the text style-transfer research (Hu et al, 2017;Fu et al, 2018;John et al, 2018), but adapt it into our scenario of syntactic modeling. We also observe that syntax and semantics are highly interwoven, and therefore further propose an adversarial reconstruction loss to regularize the syntactic and se-mantic spaces.…”

A Closer Look at Disentangling in β-VAE

“…There are other perspectives on what constitutes a disentangled representation not addressed in this paper [1], [16], including definitions not statistical in nature, instead taking into account the manifold structure and symmetry transformations in data [1], [12], [20]. Other deep learning approaches to disentangling include the adversarial setting [21]- [23]. Disentangled representations have also been studied in supervised and semi-supervised contexts [24].…”

“…We can calculate the model posterior distribution p θ (z|x) at the network optimum, eqs. (22) and (23). Using Bayes' rule we find (see Appendix B-D)…”