Neural Gaussian Copula for Variational Autoencoder

Abstract

Variational language models seek to estimate the posterior of latent variables with an approximated variational posterior. The model often assumes the variational posterior to be factorized even when the true posterior is not. The learned variational posterior under this assumption does not capture the dependency relationships over latent variables. We argue that this would cause a typical training problem called posterior collapse observed in all other variational language models. We propose Gaussian Copula Variational Autoencoder (VAE) to avert this problem. Copula is widely used to model correlation and dependencies of high-dimensional random variables, and therefore it is helpful to maintain the dependency relationships that are lost in VAE. The empirical results show that by modeling the correlation of latent variables explicitly using a neural parametric copula, we can avert this training difficulty while getting competitive results among all other VAE approaches.

Sentinel-2 Satellite Image Time-Series Land Cover Classification with Bernstein Copula Approach

“…In [25], an experiment for the classification of a single image was performed by using a classifier based on the Gaussian Copula. Moreover, some works have engaged the use of copulas in combination with the NN [26][27][28][29]. Considering the analysis of SITS data, few works already exist that exploit copulas to analyze such kinds of data.…”

Discrete Auto-regressive Variational Attention Models for Text Modeling

“…The challenge of information underrepresentation refers to the limited expressiveness of the latent space z. As shown in the left of Figure 1, current VAEs build a single latent variable z = z T based on the last hidden state of LSTM encoder [5], [6], [11], [12]. However, this is generally insufficient to summarize the input sentence [13].…”