KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support

Abstract

We study the gradient flow for a relaxed approximation to the Kullback-Leibler (KL) divergence between a moving source and a fixed target distribution. This approximation, termed the KALE (KL approximate lower-bound estimator), solves a regularized version of the Fenchel dual problem defining the KL over a restricted class of functions. When using a Reproducing Kernel Hilbert Space (RKHS) to define the function class, we show that the KALE continuously interpolates between the KL and the Maximum Mean Discrepancy (MMD). Like the MMD and other Integral Probability Metrics, the KALE remains well defined for mutually singular distributions. Nonetheless, the KALE inherits from the limiting KL a greater sensitivity to mismatch in the support of the distributions, compared with the MMD. These two properties make the KALE gradient flow particularly well suited when the target distribution is supported on a low-dimensional manifold. Under an assumption of sufficient smoothness of the trajectories, we show the global convergence of the KALE flow. We propose a particle implementation of the flow given initial samples from the source and the target distribution, which we use to empirically confirm the KALE's properties.Recent works have considered the gradient flow of Integral Probability Metrics (IPM) [42] instead of the KL, in settings where the target density is unknown and only samples from Q are available. This includes the Maximum Mean Discrepancy (MMD) [3] and the Kernelized Sobolev Discrepancy Preprint. Under review.

Outcome Assumptions and Duality Theory for Balancing Weights

“…See, for example, Dupuis and Mao (2019); Birrell et al (2020a), who generalize φ-divergences to distributions that do not have common support. This idea is applied to GANs in Song and Ermon (2020); Glaser et al (2021). Domain adaptation and causal inference.…”

Model Uncertainty and Correctability for Directed Graphical Models

“…In such cases, new divergences could be considered e.g. Wasserstein metrics already studied in the DRO literature [53,12] or their Integral Probability Metrics (IPM) generalization [55]; alternatively we can consider various interpolations of divergences and IPMs studied recently in the machine learning literature such as [30,23,6,32] and references therein. For instance, the recently introduced (f, Γ)-divergences [6] are interpolations of f -divergences and IPMs that combine advantageous features of both, such as the capability to handle heavy-tailed data (property inherited from fdivergences) and to compare non-absolutely continuous distributions (inherited from IPMs).…”