Polarized-VAE: Proximity Based Disentangled Representation Learning for Text Generation

Abstract

Learning disentangled representations of realworld data is a challenging open problem. Most previous methods have focused on either supervised approaches which use attribute labels or unsupervised approaches that manipulate the factorization in the latent space of models such as the variational autoencoder (VAE) by training with task-specific losses. In this work, we propose polarized-VAE, an approach that disentangles select attributes in the latent space based on proximity measures reflecting the similarity between data points with respect to these attributes. We apply our method to disentangle the semantics and syntax of sentences and carry out transfer experiments. Polarized-VAE outperforms the VAE baseline and is competitive with state-of-the-art approaches, while being more a general framework that is applicable to other attribute disentanglement tasks.

Unsupervised Learning of Neurosymbolic Encoders

“…We build on VAEs [29,35], a latent variable modeling framework shown to work well for learning latent representations (also called encodings/embeddings) [20,24,57,14,53,8,45,2] and capturing the generative process [36,53,46,54]. VAEs [29,35] introduce a latent variable z, an encoder q φ , a decoder p θ , and a prior distribution p on z. φ and θ are the parameters of the q and p respectively, often instantiated with neural networks.…”

Transformer-Based Neural Text Generation with Syntactic Guidance

“…For example, Hu et al (2017) generate text with specified sentiments, whereas Li et al (2018b) and Wang et al (2019a) try to transfer the sentiments or styles of the source sentences. The other track, to which our research belongs, focuses on making generated text follow a particular style or structure (Niu et al, 2017;Ficler and Goldberg, 2017;Fu et al, 2018;Iyyer et al, 2018;Li et al, 2019b;Balasubramanian et al, 2020). For instance, Niu et al (2017) constrain the output styles in neural machine translation task and impose length limitation to the summarization.…”

“…Based on the constraint source, syntactically controlled text generation models can be further divided into three groups. The first group (Chen et al, 2019b;Bao et al, 2019;Balasubramanian et al, 2020) takes sentences as syntactic exemplars. They attempt to disentangle the semantic and syntactic representations into different VAE (Kingma and Welling, 2014) latent spaces during training, and then use the exemplar to assign a prior distribution to the syntactic latent space at the inference stage.…”

Learning Disentangled Representations of Negation and Uncertainty

“…Still, no previous work has tested whether negation, uncertainty, and content can be disentangled, as linguistic theory suggests, although previous works have disentangled attributes such as syntax, semantics, and style (Balasubramanian et al, 2021;John et al, 2019;Cheng et al, 2020b;Bao et al, 2019;Hu et al, 2017;Colombo et al, 2021). To fill this gap, we aim to answer the following research questions:…”