Preventing Posterior Collapse Induced by Oversmoothing in Gaussian VAE

Takida, Yuhta; Liao, Wei-Hsiang; Lai, Chieh-Hsin; Uesaka, Toshimitsu; Takahashi, Shun; Mitsufuji, Yuki

doi:10.48550/arxiv.2102.08663

Cited by 3 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We propose using a random schedule for the variable β in order to reduce the potential effects that could be caused by the posterior collapse problem in VAEs (Lucas et al, 2019 ; Havrylov and Titov, 2020 ; Takida et al, 2021 ) and to maintain a balance with the reconstruction loss. We take a sample from a uniform distribution

for every example that we go through in the training process.…”

Section: Methodsmentioning

confidence: 99%

Feature alignment as a generative process

Farias

Maziero²

2023

Front. Artif. Intell.

View full text Add to dashboard Cite

Reversibility in artificial neural networks allows us to retrieve the input given an output. We present feature alignment, a method for approximating reversibility in arbitrary neural networks. We train a network by minimizing the distance between the output of a data point and the random output with respect to a random input. We applied the technique to the MNIST, CIFAR-10, CelebA, and STL-10 image datasets. We demonstrate that this method can roughly recover images from just their latent representation without the need of a decoder. By utilizing the formulation of variational autoencoders, we demonstrate that it is possible to produce new images that are statistically comparable to the training data. Furthermore, we demonstrate that the quality of the images can be improved by coupling a generator and a discriminator together. In addition, we show how this method, with a few minor modifications, can be used to train networks locally, which has the potential to save computational memory resources.

show abstract

for every example that we go through in the training process.…”

Section: Methodsmentioning

confidence: 99%

Feature alignment as a generative process

Farias

Maziero²

2023

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…To reduce possible effects due to the posterior collapse problem in VAEs [60][61][62] and to balance with the reconstruction loss, we propose a random schedule for the β variable. For each example during training, we sample β from a uniform distribution U (0, 1).…”

Section: Improving the Quality Of The Featuresmentioning

confidence: 99%

Feature Alignment As A Generative Process

Farias

Maziero

2022

Preprint

View full text Add to dashboard Cite

We introduce feature alignment, a technique for obtaining approximate reversibility in artificial neural networks. By means of feature extraction, we can train a neural network to learn an estimated map for its reverse process from outputs to inputs. Combined with variational autoencoders, we can generate new samples from the same statistics as the training data. Improvements of the results are obtained by using concepts from generative adversarial networks. Finally, we show that the technique can be modified for training neural networks locally, saving computational memory resources. Applying these techniques, we report results for three vision generative tasks: MNIST, CIFAR-10, and celebA.

show abstract

“…In this case, the latent variables are perturbed by adding Gaussian noises with their variance s 2 I to the encoded points. When s 2 is trained, it approaches zero (Takida et al, 2021) as the training progresses, which means that the stochastic encoding becomes almost deterministic, i.e., no perturbation.…”

Section: D1 Similarity Between Sq-vae and Conventional Vaementioning

confidence: 99%

SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization

Takida¹,

Shibuya²,

Liao³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

One noted issue of vector-quantized variational autoencoder (VQ-VAE) is that the learned discrete representation uses only a fraction of the full capacity of the codebook, also known as codebook collapse. We hypothesize that the training scheme of VQ-VAE, which involves some carefully designed heuristics, underlies this issue. In this paper, we propose a new training scheme that extends the standard VAE via novel stochastic dequantization and quantization, called stochastically quantized variational autoencoder (SQ-VAE). In SQ-VAE, we observe a trend that the quantization is stochastic at the initial stage of the training but gradually converges toward a deterministic quantization, which we call self-annealing. Our experiments show that SQ-VAE improves codebook utilization without using common heuristics. Furthermore, we empirically show that SQ-VAE is superior to VAE and VQ-VAE in vision-and speech-related tasks.

show abstract

Preventing Posterior Collapse Induced by Oversmoothing in Gaussian VAE

Cited by 3 publications

References 0 publications

Feature alignment as a generative process

Feature alignment as a generative process

Feature Alignment As A Generative Process

SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization

Contact Info

Product

Resources

About