Estimation of KL Divergence: Optimal Minimax Rate

Bu, Yuheng; Zou, Shaofeng; Liang, Y. T.; Veeravalli, Venugopal V.

doi:10.1109/tit.2018.2805844

Cited by 65 publications

(34 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We provide a comprehensive answer to this problem in this paper, thereby substantially generalizing the applicability of the approximation-based method and demonstrate the intricacy of functional estimation problems in high dimensions. Our recent work [16] presents the most up-to-date version of the general approximationbased method, which is applied to construct minimax rateoptimal estimators for the KL divergence (also see Bu et al [17]), χ 2 -divergence, and the squared Hellinger distance. The effective sample size enlargement phenomenon holds in all these cases as well.…”

Section: B Approximation-based Methodsmentioning

confidence: 99%

Minimax estimation of the L<inf>1</inf> distance

Jiao

Han

Weissman

2016

2016 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

We consider the problem of estimating the L1 distance between two discrete probability measures P and Q from empirical data in a nonasymptotic and large alphabet setting. When Q is known and one obtains n samples from P , we show that for every Q, the minimax rate-optimal estimator with n samples achieves performance comparable to that of the maximum likelihood estimator (MLE) with n ln n samples. When both P and Q are unknown, we construct minimax rateoptimal estimators whose worst case performance is essentially that of the known Q case with Q being uniform, implying that Q being uniform is essentially the most difficult case. The effective sample size enlargement phenomenon, identified in Jiao et al. (2015), holds both in the known Q case for every Q and the Q unknown case. However, the construction of optimal estimators for P − Q 1 requires new techniques and insights beyond the approximation-based method of functional estimation in .

show abstract

Section: B Approximation-based Methodsmentioning

confidence: 99%

Minimax estimation of the L<inf>1</inf> distance

Jiao

Han

Weissman

2016

2016 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

show abstract

“…Classically, there does not exist any consistent estimator that guarantees asymptotically small error over the set of all pairs of distributions [14,27]. These two papers then consider pairs of distributions with bounded probability ratios specified by a function f : N → R + , namely all pairs of distributions in the set as follows:…”

Section: Application: Kl Divergence Estimationmentioning

confidence: 99%

“…Denote the number of samples from p and q to be M p and M q , respectively. References [14,27] shows that classically, D KL (p q) can be approximated within constant additive error with high success probability if and only if M p = Ω( n log n ) and M q = Ω( nf (n) log n ). Quantumly, we are given unitary oraclesÔ p andÔ q defined by (1.7).…”

Section: Application: Kl Divergence Estimationmentioning

confidence: 99%

Quantum Query Complexity of Entropy Estimation

2019

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Estimation of Shannon and Rényi entropies of unknown discrete distributions is a fundamental problem in statistical property testing and an active research topic in both theoretical computer science and information theory. Tight bounds on the number of samples to estimate these entropies have been established in the classical setting, while little is known about their quantum counterparts. In this paper, we give the first quantum algorithms for estimating α-Rényi entropies (Shannon entropy being 1-Renyi entropy). In particular, we demonstrate a quadratic quantum speedup for Shannon entropy estimation and a generic quantum speedup for α-Rényi entropy estimation for all α ≥ 0, including a tight bound for the collision-entropy (2-Rényi entropy). We also provide quantum upper bounds for extreme cases such as the Hartley entropy (i.e., the logarithm of the support size of a distribution, corresponding to α = 0) and the min-entropy case (i.e., α = +∞), as well as the Kullback-Leibler divergence between two distributions. Moreover, we complement our results with quantum lower bounds on α-Rényi entropy estimation for all α ≥ 0.Our approach is inspired by the pioneering work of Bravyi, Harrow, and Hassidim (BHH) [13] on quantum algorithms for distributional property testing, however, with many new technical ingredients. For Shannon entropy and 0-Rényi entropy estimation, we improve the performance of the BHH framework, especially its error dependence, using Montanaro's approach to estimating the expected output value of a quantum subroutine with bounded variance [41] and giving a fine-tuned error analysis. For general α-Rényi entropy estimation, we further develop a procedure that recursively approximates α-Rényi entropy for a sequence of αs, which is in spirit similar to a cooling schedule in simulated annealing. For special cases such as integer α ≥ 2 and α = +∞ (i.e., the min-entropy), we reduce the entropy estimation problem to the α-distinctness and the log n -distinctness problems, respectively. We exploit various techniques to obtain our lower bounds for different ranges of α, including reductions to (variants of) existing lower bounds in quantum query complexity as well as the polynomial method inspired by the celebrated quantum lower bound for the collision problem.

show abstract

“…Although we focus on Bayesian machinery, one can estimate D-probabilities using any method that estimates KL(f 0 , f j ). Substantial work has focused on estimating the Kullback-Leibler divergence between two unknown densities based on samples from these densities (Leonenko et al, 2008;Pérez-Cruz, 2008;Bu et al, 2018). Our setting is somewhat different, but the local likelihood methods of Lee and Park (2006) and the Bayesian approach of Viele (2007) can potentially be used, among others.…”

Section: Estimation Of Model Weights: D-probabilitiesmentioning

confidence: 99%

Comparing and Weighting Imperfect Models Using D-Probabilities

Dunson

2019

Journal of the American Statistical Association

View full text Add to dashboard Cite

We propose a new approach for assigning weights to models using a divergence-based method (D-probabilities), relying on evaluating parametric models relative to a nonparametric Bayesian reference using Kullback-Leibler divergence. D-probabilities are useful in goodnessof-fit assessments, in comparing imperfect models, and in providing model weights to be used in model aggregation. D-probabilities avoid some of the disadvantages of Bayesian model probabilities, such as large sensitivity to prior choice, and tend to place higher weight on a greater diversity of models. In an application to linear model selection against a Gaussian process reference, we provide simple analytic forms for routine implementation and show that D-probabilities automatically penalize model complexity. Some asymptotic properties are described, and we provide interesting probabilistic interpretations of the proposed model weights. The framework is illustrated through simulation examples and an ozone data application.

show abstract

Estimation of KL Divergence: Optimal Minimax Rate

Cited by 65 publications

References 25 publications

Minimax estimation of the L<inf>1</inf> distance

Minimax estimation of the L<inf>1</inf> distance

Quantum Query Complexity of Entropy Estimation

Comparing and Weighting Imperfect Models Using D-Probabilities

Contact Info

Product

Resources

About