Estimation of Nonlinear Functionals of Densities With Confidence

Abstract

This paper introduces a class of k-nearest neighbor (k-NN) estimators called bipartite plug-in (BPI) estimators for estimating integrals of non-linear functions of a probability density, such as Shannon entropy and Rényi entropy. The density is assumed to be smooth, have bounded support, and be uniformly bounded from below on this set. Unlike previous k-NN estimators of non-linear density functionals, the proposed estimator uses data-splitting and boundary correction to achieve lower mean square error. Specifically, we assume that T i.i.d. samples Xi ∈ R d from the density are split into two pieces of cardinality M and N respectively, with M samples used for computing a k-nearest-neighbor density estimate and the remaining N samples used for empirical estimation of the integral of the density functional.By studying the statistical properties of k-NN balls, explicit rates for the bias and variance of the BPI estimator are derived in terms of the sample size, the dimension of the samples and the underlying probability distribution. Based on these results, it is possible to specify optimal choice of tuning parameters M/T , k for maximizing the rate of decrease of the mean square error (MSE). The resultant optimized BPI estimator converges faster and achieves lower mean squared error than previous k-NN entropy estimators. In addition, a central limit theorem is established for the BPI estimator that allows us to specify tight asymptotic confidence intervals.

Performance-driven Multimodality Sensor Fusion

“…For a broad class of density-plug-in estimators, that includes the common kernel density and k nearest neighbor (kNN) plug-in estimators, we have developed a generally applicable theory that gives analytical closed-form expressions for asymptotic bias and MSE in terms of the sample size, the dimension of the feature space, and the underlying feature probability distribution. These results appear in the technical report [6] co-authored by co-PI's Hero and Raich and the supported University of Michigan graduate student Kumar Sricharan. This report incorporates comparisons to state of the art divergence and entropy estimation algorithms and is an extension of the report cited in last year's progress report.…”

“…We have obtained the (to-date) sharpest asymptotic expressions for estimator bias, variance and a CLT for a wide class of information divergence estimators [6], [7]. The utility of these expressions is that they can be used to optimize over tuning parameters of the fusion criterion, thereby circumventing the need for manual parameter tuning.…”

k-nearest neighbor estimation of entropies with confidence

“…This in turn can be used to predict and optimize performance in applications like structure discovery in graphical models and dimension estimation for support sets of low intrinsic dimension. See [20] for more details on these applications.…”

“…to show that convergence in distribution to N (0, 1) holds in our case as both N and M get large. These ideas are rigorously treated in Appendix E, [20].…”

“…Conclusions are given in Section VI. Additional details on proofs and results are given in our technical report [20].…”