Statistical estimation of the Kullback-Leibler divergence

Abstract

Wide conditions are provided to guarantee asymptotic unbiasedness and L 2 -consistency of the introduced estimates of the Kullback -Leibler divergence for probability measures in R d having densities w.r.t. the Lebesgue measure. These estimates are constructed by means of two independent collections of i.i.d. observations and involve the specified k-nearest neighbor statistics. In particular, the established results are valid for estimates of the Kullback -Leibler divergence between any two Gaussian measures in R d with nondegenerate covariance matrices. As a byproduct we obtain new statements concerning the Kozachenko-Leonenko estimators of the Shannon differential entropy.