Two-sample homogeneity tests based on divergence measures

Wornowizki, Max; Fried, Roland

doi:10.1007/s00180-015-0633-3

Cited by 13 publications

(12 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, the two-sample test statistic between Ŝ and S measures the fidelity of the samples Ŝ produced by the generative model. The use of two-sample tests to evaluate the sample quality of generative models include the pioneering work of Box (1980), the use of Maximum Mean Discrepancy (MMD) criterion (Bengio et al, 2013;Dziugaite et al, 2015;Lloyd & Ghahramani, 2015;Bounliphone et al, 2015;Sutherland et al, 2016), and the connections to density-ratio estimation (Kanamori et al, 2010;Wornowizki & Fried, 2016;Menon & Ong, 2016;Mohamed & Lakshminarayanan, 2016).…”

Section: Two-sample Testingmentioning

confidence: 99%

Revisiting Classifier Two-Sample Tests

Lopez-Paz,

Oquab

2016

Preprint

View full text Add to dashboard Cite

The goal of two-sample tests is to assess whether two samples, S P ∼ P n and S Q ∼ Q m , are drawn from the same distribution. Perhaps intriguingly, one relatively unexplored method to build two-sample tests is the use of binary classifiers. In particular, construct a dataset by pairing the n examples in S P with a positive label, and by pairing the m examples in S Q with a negative label. If the null hypothesis "P = Q" is true, then the classification accuracy of a binary classifier on a held-out subset of this dataset should remain near chance-level. As we will show, such Classifier Two-Sample Tests (C2ST) learn a suitable representation of the data on the fly, return test statistics in interpretable units, have a simple null distribution, and their predictive uncertainty allow to interpret where P and Q differ. The goal of this paper is to establish the properties, performance, and uses of C2ST. First, we analyze their main theoretical properties. Second, we compare their performance against a variety of state-of-the-art alternatives. Third, we propose their use to evaluate the sample quality of generative models with intractable likelihoods, such as Generative Adversarial Networks (GANs). Fourth, we showcase the novel application of GANs together with C2ST for causal discovery.

show abstract

Section: Two-sample Testingmentioning

confidence: 99%

Revisiting Classifier Two-Sample Tests

Lopez-Paz,

Oquab

2016

Preprint

View full text Add to dashboard Cite

show abstract

“…In information theory, importance weight estimation (where it is sometimes known as Radon-Nikodym derivative estimation), is applied to estimate various divergences such as the KL divergence and Renyi divergences between the distributions P and R [NWJ07, NWJ10, YSK + 13, WKV05, WKV09]. These divergence measures themselves find several applications, such as two-sample tests for distinguishing between two distributions [WF16] and for independence testing [SSSK08].…”

Section: Introductionmentioning

confidence: 99%

Multicalibrated Partitions for Importance Weights

Gopalan¹,

Reingold²,

Sharan³

et al. 2021

Preprint

View full text Add to dashboard Cite

The ratio between the probability that two distributions R and P give to points x are known as importance weights or propensity scores and play a fundamental role in many different fields, most notably, statistics and machine learning. Among its applications, importance weights are central to domain adaptation, anomaly detection, and estimations of various divergences such as the KL divergence and Renyi divergences between R and P , which in turn have numerous applications. We consider the common setting where R and P are only given through samples from each distribution. The vast literature on estimating importance weights is either heuristic, or makes strong assumptions about R and P or on the importance weights themselves. Indeed, relying on cryptographic assumptions, we show the impossibility of efficiently computing pointwise accurate importance weights.In this paper, we explore a computational perspective to the estimation of importance weights, which factors in the limitations and possibilities obtainable with bounded computational resources. We significantly strengthen previous work that use the MaxEntropy approach, that define the importance weights based on a distribution Q closest to P , that looks the same as R on every set C ∈ C, where C may be a huge collection of sets. We show that the MaxEntropy approach may fail to assign high average scores to sets C ∈ C, even when the average of ground truth weights for the set is evidently large. We similarly show that it may overestimate the average scores to sets C ∈ C. We therefore formulate Sandwiching bounds as a notion of set-wise accuracy for importance weights. We study these bounds to show that they capture natural completeness and soundness requirements from the weights and are appealing from the point of view of accuracy and fairness in heterogeneous populations. We present an efficient algorithm that under standard learnability assumptions computes weights which satisfy these bounds.Our techniques rely on a new notion of multicalibrated partitions of the domain of the distributions. While being a relatively small collection of disjoint sets, such partitions reflect, in a well-defined sense, the complexity of the much larger collection of arbitrarily intersecting sets in C. Stratifying the domain based on multicalibrated partitions implies our computational objectives for importance weights and appear to be useful objects in their own right.

show abstract

“…Training classifier provides an estimator of density ratio, as has been pointed out in [27] and in the formulation of learning generative models [28]. While distribution divergence estimation has been studied and used for two-sample problems [18,19,37,40], the use of neural network as a divergence estimator for two-sample testing was less investigated. In terms of theoretical guarantee of test power, the analysis in [26] assumes a non-zero population test statistic under H 1 which is not specified, along with other approximations.…”

Section: Related Workmentioning

confidence: 99%

“…Many existing tests are based on certain estimators of a distance or divergence between p and q. Important examples include Maximum Mean Discrepancy (MMD), especially kernel-based MMD [1,13] and distance of Reproducing Kernel Hilbert Space Mean Embedding [9,16], divergence based methods which may involve non-parametric estimation of density difference or density ratio [19,36,37,40]. While these methods have been intensively studied and theoretically well-understood, the application is often restricted to data of small dimensionality and/or small sample size, or certain specific classes of densities p and q, due to model and computational limitations.…”

Section: Introductionmentioning

confidence: 99%

Classification Logit Two-sample Testing by Neural Networks

Cheng,

Cloninger

2019

Preprint

View full text Add to dashboard Cite

The recent success of generative adversarial networks and variational learning suggests training a classifier network may work well in addressing the classical two-sample problem. Network-based tests have the computational advantage that the algorithm scales to large samples. This paper proposes to use the difference of the logit of a trained neural network classifier evaluated on the two finite samples as the test statistic. Theoretically, we prove the testing power to differentiate two smooth densities given that the network is sufficiently parametrized, by comparing the learned logit function to the log ratio of the densities, the latter maximizing the population training objective. When the two densities lie on or near to low-dimensional manifolds embedded in possibly high-dimensional space, the needed network complexity is reduced to only depending on the intrinsic manifold geometry. In experiments, the method demonstrates better performance than previous network-based tests which use the classification accuracy as the test statistic, and compares favorably to certain kernel maximum mean discrepancy (MMD) tests on synthetic and hand-written digits datasets.

show abstract

Two-sample homogeneity tests based on divergence measures

Cited by 13 publications

References 23 publications

Revisiting Classifier Two-Sample Tests

Revisiting Classifier Two-Sample Tests

Multicalibrated Partitions for Importance Weights

Classification Logit Two-sample Testing by Neural Networks

Contact Info

Product

Resources

About