Variational Diffusion Models

Abstract

Diffusion-based generative models have demonstrated a capacity for perceptually impressive synthesis, but can they also be great likelihood-based models? We answer this in the affirmative, and introduce a family of diffusion-based generative models that obtain state-of-the-art likelihoods on standard image density estimation benchmarks. Unlike other diffusion-based models, our method allows for efficient optimization of the noise schedule jointly with the rest of the model. We show that the variational lower bound (VLB) simplifies to a remarkably short expression in terms of the signal-to-noise ratio of the diffused data, thereby improving our theoretical understanding of this model class. Using this insight, we prove an equivalence between several models proposed in the literature. In addition, we show that the continuous-time VLB is invariant to the noise schedule, except for the signal-to-noise ratio at its endpoints. This enables us to learn a noise schedule that minimizes the variance of the resulting VLB estimator, leading to faster optimization. Combining these advances with architectural improvements, we obtain state-of-the-art likelihoods on image density estimation benchmarks, outperforming autoregressive models that have dominated these benchmarks for many years, with often significantly faster optimization. In addition, we show how to turn the model into a bits-back compression scheme, and demonstrate lossless compression rates close to the theoretical optimum. * Equal contribution.Preprint. Under review.

Density Ratio Estimation via Infinitesimal Classification

“…In our preliminary experiments, we found that a naive implementation of Proposition 2 led to unstable training due to the high variance in the objective across the different timescales t. This finding is in accordance with recent work on diffusion probabilistic models (Nichol and Dhariwal, 2021;Kingma et al, 2021;Song et al, 2021a), which emphasize the critical role of applying the proper weighting function λptq rather than randomly sampling t " Ur0, 1s. This problem is also exacerbated by the fact that our training objective requires backpropagating through the score network s time θ px, tq with respect to t, which can cause training to diverge or progress extremely slowly for certain design choices.…”

“…Additionally, DRE-8 takes longer to converge as qpxq becomes further apart from ppxq, though this speaks to the challenging nature of the DRE problem as a whole. It would be interesting to investigate whether there is a time-dependent function λptq such that the time score matching loss corresponds to the maximum likelihood training of a binary classifier (Song et al, 2021a;Kingma et al, 2021). Additionally, exploring optimal integration paths between ppxq and qpxq would be exciting future work.…”

“…Drawing inspiration from Nichol and Dhariwal (2021); Song et al (2021a); Kingma et al (2021), we learn an importance weighting scheme of the distribution over timescales pptq. We optimize the following importanceweighted objective rather than Proposition 2:…”

“…When designing the time score network architecture for more complex datasets, we found that both sinusoidal positional embeddings (Ho et al, 2020) and Fourier embeddings (Tancik et al, 2020;Song et al, 2021b;Kingma et al, 2021) commonly used in the literature led to training instabilities when computing B Bt s time θ px, tq. We hypothesize that this is due to the periodic nature of the sine and cosine functions, causing gradient information with respect to t to oscillate during training.…”

TFDPM: Attack detection for cyber-physical systems with diffusion probabilistic models

“…In addition, with the signal-to-noise ratio implemented into the training objective in Eq. 20, we can get L 2Ns t − L Ns < 0 [48]. Therefore, large diffusion steps can help further improve the performance.…”