Minimax estimation of the L&lt;inf&gt;1&lt;/inf&gt; distance

Jiao, Jiantao; Han, Yanjun; Weissman, Tsachy

doi:10.1109/isit.2016.7541399

Cited by 44 publications

(76 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Remark 2. The results on R EST (D, L, n) and R PLU (D, L, n) follow from [5]. The key contribution of this paper is the solution of R ACH (D, L, n), whose upper and lower bounds prove to be non-trivial.…”

Section: ) Effective Sample Size Enlargementmentioning

confidence: 62%

See 1 more Smart Citation

Estimating the Fundamental Limits is Easier Than Achieving the Fundamental Limits

Jiao

Han

Fischer-Hwang

et al. 2019

IEEE Trans. Inform. Theory

Self Cite

View full text Add to dashboard Cite

We show through case studies that it is easier to estimate the fundamental limits of data processing than to construct explicit algorithms to achieve those limits. Focusing on binary classification, data compression, and prediction under logarithmic loss, we show that in the finite space setting, when it is possible to construct an estimator of the limits with vanishing error with n samples, it may require at least n ln n samples to construct an explicit algorithm to achieve the limits.

show abstract

Section: ) Effective Sample Size Enlargementmentioning

confidence: 62%

“…Now we consider the second statement. We used the classical splitting operation [5] to represent random variable X as…”

Section: A Proof Of Lemmamentioning

confidence: 99%

Estimating the Fundamental Limits is Easier Than Achieving the Fundamental Limits

Jiao

Han

Fischer-Hwang

et al. 2019

IEEE Trans. Inform. Theory

Self Cite

View full text Add to dashboard Cite

show abstract

“…Simple as it may sound, this methodology has a few drawbacks and ambiguities. In our recent work [24], we applied this general recipe to the estimation of 1 distance between two discrete distributions, where this recipe proves to be inadequate. In the estimation of the 1 distance, a bivariate function f (x, y) = |x − y| which is non-analytic in a segment was considered, which is completely different from the previous studies [1], [28]- [31] where a univariate function analytic everywhere except a point is always taken into consideration.…”

Section: A Background and Main Resultsmentioning

confidence: 99%

Minimax estimation of discrete distributions

Han

Jiao

Weissman

2015

2015 IEEE International Symposium on Information Theory (ISIT)

Self Cite

View full text Add to dashboard Cite

We refine the general methodology in [1] for the construction and analysis of essentially minimax estimators for a wide class of functionals of finite dimensional parameters, and elaborate on the case of discrete distributions with support size S comparable with the number of observations n. Specifically, we determine the "smooth" and "non-smooth" regimes based on the confidence set and the smoothness of the functional. In the "non-smooth" regime, we apply an unbiased estimator for a suitable polynomial approximation of the functional. In the "smooth" regime, we construct a general version of the bias-corrected Maximum Likelihood Estimator (MLE) based on Taylor expansion.We apply the general methodology to the problem of estimating the KL divergence between two discrete probability measures P and Q from empirical data in a non-asymptotic and possibly large alphabet setting. We construct minimax rate-optimal estimators for D(P Q) when the likelihood ratio is upper bounded by a constant which may depend on the support size, and show that the performance of the optimal estimator with n samples is essentially that of the MLE with n ln n samples. Our estimator is adaptive in the sense that it does not require the knowledge of the support size nor the upper bound on the likelihood ratio. We show that the general methodology results in minimax rate-optimal estimators for other divergences as well, such as the Hellinger distance and the χ 2 -divergence. Our approach refines the Approximation methodology recently developed for the construction of near minimax estimators of functionals of high-dimensional parameters, such as entropy, Rényi entropy, mutual information and 1 distance in large alphabet settings, and shows that the effective sample size enlargement phenomenon holds significantly more widely than previously established.

show abstract

“…However, it is challenging to find the explicit form of the two-dimensional polynomial approximation for estimating KL divergence as shown in [25]. However, for some problems it is still worth exploring the two-dimensional approximation directly, as shown in [26], where no onedimensional approximation can achieve the minimax rate.…”

Section: Minimax Upper Bound Via Optimal Estimatormentioning

confidence: 99%

Estimation of KL Divergence: Optimal Minimax Rate

Zou

Liang

et al. 2018

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

The problem of estimating the Kullback-Leibler divergence D(P Q) between two unknown distributions P and Q is studied, under the assumption that the alphabet size k of the distributions can scale to infinity. The estimation is based on m independent samples drawn from P and n independent samples drawn from Q. It is first shown that there does not exist any consistent estimator that guarantees asymptotically small worstcase quadratic risk over the set of all pairs of distributions. A restricted set that contains pairs of distributions, with density ratio bounded by a function f (k) is further considered. An augmented plug-in estimator is proposed, and its worst-case quadratic risk is shown to be within a constant factor of (n , if m and n exceed a constant factor of k and kf (k), respectively. Moreover, the minimax quadratic risk is characterized to be within a constant factor of ( k m log k + kf (k)n , if m and n exceed a constant factor of k/ log(k) and kf (k)/ log k, respectively. The lower bound on the minimax quadratic risk is characterized by employing a generalized Le Cam's method. A minimax optimal estimator is then constructed by employing both the polynomial approximation and the plug-in approaches.

show abstract

Minimax estimation of the L<inf>1</inf> distance

Cited by 44 publications

References 30 publications

Estimating the Fundamental Limits is Easier Than Achieving the Fundamental Limits

Estimating the Fundamental Limits is Easier Than Achieving the Fundamental Limits

Minimax estimation of discrete distributions

Estimation of KL Divergence: Optimal Minimax Rate

Contact Info

Product

Resources

About