FlowPrior: Learning Expressive Priors for Latent Variable Sentence Models

Ding, Xiaoan; Gimpel, Kevin

doi:10.18653/v1/2021.naacl-main.259

Cited by 3 publications

(10 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Expressive Prior and Posterior In VAEs VAEs usually employ simple Gaussian distribution as the prior and spherical Gaussian distributions with diagonal co-variance matrices as the variational posteriors (Higgins et al, 2017;He et al, 2019;Fu et al, 2019). Such predefined forms in traditional formulations hinder VAEs from the expressivity of the model (Ding and Gimpel, 2021), thus further inducing the posterior collapse (Fang et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

“…To improve the representation capacity, some efforts try more expressive priors. MoG-VAE (Ding and Gimpel, 2021) considers a uniform mixture of Gaussians as the prior, Vamp-VAE (Tomczak and Welling, 2018) introduces "Variational Mixture of Posteriors" prior (VampPrior). APo-VAE (Dai et al, 2021) adopts VampPrior to learn a hyperbolic latent space.…”

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…posteriors (Ding and Gimpel, 2021), which generally follow Gaussian distribution and spherical Gaussian distributions with diagonal co-variance matrices, respectively (Higgins et al, 2017;He et al, 2019;Li et al, 2019a). Such predefined forms would hinder VAEs from larger optimization space (Fang et al, 2019), thus restricting the expressivity of the model (Ding and Gimpel, 2021) and further leading to the posterior collapse (Fang et al, 2019). Therefore, a potential solution is to try more expressive distribution forms for priors and variational posteriors to improve the representation capacity (Fang et al, 2019;Tomczak and Welling, 2018;Ding and Gimpel, 2021).…”

Section: Introductionmentioning

confidence: 99%

“…Such predefined forms would hinder VAEs from larger optimization space (Fang et al, 2019), thus restricting the expressivity of the model (Ding and Gimpel, 2021) and further leading to the posterior collapse (Fang et al, 2019). Therefore, a potential solution is to try more expressive distribution forms for priors and variational posteriors to improve the representation capacity (Fang et al, 2019;Tomczak and Welling, 2018;Ding and Gimpel, 2021).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Controlled Text Generation Using Dictionary Prior in Variational Autoencoders

Fang¹,

Li²,

Shang³

et al. 2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

While variational autoencoders (VAEs) have been widely applied in text generation tasks, they are troubled by two challenges: insufficient representation capacity and poor controllability. The former results from the posterior collapse and restrictive assumption, which impede better representation learning. The latter arises as continuous latent variables in traditional formulations hinder VAEs from interpretability and controllability. In this paper, we propose Dictionary Prior (DPrior), a new data-driven prior that enjoys the merits of expressivity and controllability. To facilitate controlled text generation with DPrior, we propose to employ contrastive learning to separate the latent space into several parts. Extensive experiments on both language modeling and controlled text generation demonstrate the effectiveness of the proposed approach.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Controlled Text Generation Using Dictionary Prior in Variational Autoencoders

Fang¹,

Li²,

Shang³

et al. 2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

show abstract

A Tale of Two Latent Flows: Learning Latent Space Normalizing Flow with Short-Run Langevin Flow for Approximate Inference

Xie

Zhu

et al. 2023

AAAI

View full text Add to dashboard Cite

We study a normalizing flow in the latent space of a top-down generator model, in which the normalizing flow model plays the role of the informative prior model of the generator. We propose to jointly learn the latent space normalizing flow prior model and the top-down generator model by a Markov chain Monte Carlo (MCMC)-based maximum likelihood algorithm, where a short-run Langevin sampling from the intractable posterior distribution is performed to infer the latent variables for each observed example, so that the parameters of the normalizing flow prior and the generator can be updated with the inferred latent variables. We show that, under the scenario of non-convergent short-run MCMC, the finite step Langevin dynamics is a flow-like approximate inference model and the learning objective actually follows the perturbation of the maximum likelihood estimation (MLE). We further point out that the learning framework seeks to (i) match the latent space normalizing flow and the aggregated posterior produced by the short-run Langevin flow, and (ii) bias the model from MLE such that the short-run Langevin flow inference is close to the true posterior. Empirical results of extensive experiments validate the effectiveness of the proposed latent space normalizing flow model in the tasks of image generation, image reconstruction, anomaly detection, supervised image inpainting and unsupervised image recovery.

show abstract

A Tale of Two Latent Flows: Learning Latent Space Normalizing Flow with Short-run Langevin Flow for Approximate Inference

Xie¹,

Yaxuan²,

Xu³

et al. 2023

Preprint

View full text Add to dashboard Cite

We study a normalizing flow in the latent space of a topdown generator model, in which the normalizing flow model plays the role of the informative prior model of the generator. We propose to jointly learn the latent space normalizing flow prior model and the top-down generator model by a Markov chain Monte Carlo (MCMC)-based maximum likelihood algorithm, where a short-run Langevin sampling from the intractable posterior distribution is performed to infer the latent variables for each observed example, so that the parameters of the normalizing flow prior and the generator can be updated with the inferred latent variables. We show that, under the scenario of non-convergent short-run MCMC, the finite step Langevin dynamics is a flow-like approximate inference model and the learning objective actually follows the perturbation of the maximum likelihood estimation (MLE). We further point out that the learning framework seeks to (i) match the latent space normalizing flow and the aggregated posterior produced by the short-run Langevin flow, and (ii) bias the model from MLE such that the short-run Langevin flow inference is close to the true posterior. Empirical results of extensive experiments validate the effectiveness of the proposed latent space normalizing flow model in the tasks of image generation, image reconstruction, anomaly detection, supervised image inpainting and unsupervised image recovery.

show abstract

FlowPrior: Learning Expressive Priors for Latent Variable Sentence Models

Cited by 3 publications

References 21 publications

Controlled Text Generation Using Dictionary Prior in Variational Autoencoders

Controlled Text Generation Using Dictionary Prior in Variational Autoencoders

A Tale of Two Latent Flows: Learning Latent Space Normalizing Flow with Short-Run Langevin Flow for Approximate Inference

A Tale of Two Latent Flows: Learning Latent Space Normalizing Flow with Short-run Langevin Flow for Approximate Inference

Contact Info

Product

Resources

About