Adversarial Decomposition of Text Representation

Abstract

In this paper, we present a method for adversarial decomposition of text representation. This method can be used to decompose a representation of an input sentence into several independent vectors, each of them responsible for a specific aspect of the input sentence. We evaluate the proposed method on two case studies: the conversion between different social registers and diachronic language change. We show that the proposed method is capable of fine-grained controlled change of these aspects of the input sentence. It is also learning a continuous (rather than categorical) representation of the style of the sentence, which is more linguistically realistic. The model uses adversarial-motivational training and includes a special motivational loss, which acts opposite to the discriminator and encourages a better decomposition. Furthermore, we evaluate the obtained meaning embeddings on a downstream task of paraphrase detection and show that they significantly outperform the embeddings of a regular autoencoder.

Style Transfer for Texts: Retrain, Report Errors, Compare with Rewrites

“…Such a component is proposed by who demonstrate that decomposition of style and content could be improved with an auxiliary multi-task for label prediction and adversarial objective for bag-of-words prediction. Romanov et al (2018) also introduces a dedicated component to control semantic aspects of latent representations and an adversarial-motivational training that includes a special motivational loss to encourage a better decomposition. Speaking about preservation of semantics one also has to mention works on paraphrase systems, see, for example (Prakash et al, 2016;Gupta et al, 2018;Roy and Grangier, 2019).…”

“…Architecture with additional discriminator, shifted autoencoder (SAE) with additional cosine losses, and a combination of these two architectures measured after five re-runs outperform the baseline by (Hu et al, 2017a) as well as other state of the art models. Results of (Romanov et al, 2018) are not displayed due to the absence of self-reported BLEU scores…”

Style transfer with variational autoencoders is a promising approach to RNA-Seq data harmonization and analysis

“…Learning the representations independent of domain with the help of discriminator using gradient reversal layer was proposed in the study by Gatys et al (2015) . The adversarial decomposition strategy was successfully applied to style transfer of texts, for example, in this work ( Romanov et al , 2018 ). In our work, we also used cycle-consistency loss for style transfer, which was proposed in the study by Zhu et al (2017) .…”

Decomposing Textual Information For Style Transfer

“…A dedicated component that controls semantic component of the latent representation is proposed by who demonstrate that decomposition of style and content could be improved with an auxiliary multi-task for label prediction and adversarial objective for a bag-of-words prediction. Romanov et al (2018) also introduce a dedicated component to control semantic aspects of latent representations and an adversarial-motivational training that includes a special motivational loss to encourage a better decomposition.…”