Learning deep kernels for exponential family densities

Wenliang, Li Kevin; Sutherland, Danica J.; Strathmann, Heiko; Gretton, Arthur

doi:10.48550/arxiv.1811.08357

Cited by 8 publications

(11 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To obtain consistent estimator from the kernel exponential family the authors of [14] used denoising score matching with Taylor series expansion. This results in an additional regularization term in the loss function that penalizes second derivatives of the model.…”

Section: Kernel Exponential Familymentioning

confidence: 99%

“…At the end of the training we estimate the normalizing constant via importance sampling as was proposed in [14]. It should be noted that in the case of uniform base density normalization could not be estimated properly due to the unknown data support measure.…”

Section: Rff For Denoising Score Matchingmentioning

confidence: 99%

“…To approach the computational complexity issue [13,14] propose to use Nyström-type approximation of the kernel function. Alternatively, Random Fourier Features (RFF) [15] embedding can be considered.…”

Section: Introductionmentioning

confidence: 99%

“…The first one is an oscillating behaviour in the tails of the distribution [12]. The second problem is poor convergence in the case of disjoint support (consistency could not be guaranteed) or in the areas where density value is close to zero [14]. Inconsistency explains low approximation accuracy in regions of almost zero density.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Denoising Score Matching with Random Fourier Features

Olga,

Kapushev,

Burnaev

et al. 2021

Preprint

View full text Add to dashboard Cite

The density estimation is one of the core problems in statistics. Despite this, existing techniques like maximum likelihood estimation are computationally inefficient due to the intractability of the normalizing constant. For this reason an interest to score matching has increased being independent on the normalizing constant. However, such estimator is consistent only for distributions with the full space support. One of the approaches to make it consistent is to add noise to the input data which is called Denoising Score Matching.In this work we derive analytical expression for the Denoising Score matching using the Kernel Exponential Family as a model distribution. The usage of the kernel exponential family is motivated by the richness of this class of densities. To tackle the computational complexity we use Random Fourier Features based approximation of the kernel function. The analytical expression allows to drop additional regularization terms based on the higher-order derivatives as they are already implicitly included. Moreover, the obtained expression explicitly depends on the noise variance, so the validation loss can be straightforwardly used to tune the noise level. Along with benchmark experiments, the model was tested on various synthetic distributions to study the behaviour of the model in different cases. The empirical study shows comparable quality to the competing approaches, while the proposed method being computationally faster. The latter one enables scaling up to complex high-dimensional data.

show abstract

Section: Kernel Exponential Familymentioning

confidence: 99%

Section: Rff For Denoising Score Matchingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Denoising Score Matching with Random Fourier Features

Olga,

Kapushev,

Burnaev

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Stein's method [5] provides an elegant probabilistic tool for comparing distributions based on Stein operators acting on a broad class of test functions, which has been used to tackle various problems in statistical inference, random graph theory, and computational biology. Modern machine learning tasks, such as density estimations [29,47,57], model criticisms [34,48,55], or generative modellings [20,40], may extensively involve the modelling and learning with intractable densities, where the normalisation constant (or partition function) is unable to be obtained in closed form. Stein operators may only require access to the distributions through the differential (or difference) of the log density functions (or mass functions), which avoids the knowledge of the normalisation constant.…”

Section: Introductionmentioning

confidence: 99%

Standardisation-function Kernel Stein Discrepancy: A Unifying View on Kernel Stein Discrepancy Tests for Goodness-of-fit

Xu¹

2021

Preprint

View full text Add to dashboard Cite

Non-parametric goodness-of-fit testing procedures based on kernel Stein discrepancies (KSD) are promising approaches to validate general unnormalised distributions in various scenarios. Existing works have focused on studying optimal kernel choices to boost test performances. However, the Stein operators are generally non-unique, while different choices of Stein operators can also have considerable effect on the test performances. In this work, we propose a unifying framework, the generalised kernel Stein discrepancy (GKSD), to theoretically compare and interpret different Stein operators in performing the KSD-based goodness-of-fit tests. We derive explicitly that how the proposed GKSD framework generalises existing Stein operators and their corresponding tests. In addition, we show that GKSD framework can be used as a guide to develop kernel-based non-parametric goodness-of-fit tests for complex new data scenarios, e.g. truncated distributions or compositional data. Experimental results demonstrate that the proposed tests control type-I error well and achieve higher test power than existing approaches, including the test based on maximum-mean-discrepancy (MMD).Preprint. Under review.

show abstract