Statistical estimation of the Shannon entropy

Bulinski, Alexander; Denis, Dimitrov,

doi:10.48550/arxiv.1801.02050

Cited by 2 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that our estimates do not employ the well-known Kozachenko-Leonenko statistics [22] used for estimation of the unconditional Shannon entropy. Under simple assumptions (cf., e.g., [5], [10], [14], [18], [36]) we establish the asymptotic unbiasedness and L 2 -consistency of our estimates when the sample size tends to infinity. An interest in the study of conditional entropy is explained as follows.…”

Section: Introductionmentioning

confidence: 90%

“…There are a number of papers devoted to various estimates of (unconditional) entropy. In this regard we indicate the recent work [10] where the estimates of the Shannon differential entropy are studied and where one can find further references. The scheme (4) under consideration comprises the famous logistic model widely used in the classification problems (see, e.g., [24]).…”

Section: Introductionmentioning

confidence: 95%

“…Let X ∼ N (a, Σ) where a ∈ R d and Σ > 0. It is easily seen that, for any ε > 0, one has E| log f X (X)| ε < ∞ (see, e.g., [10]). Since, for x ∈ R d and y ∈ M ,…”

Section: Applying the Hoeffding Inequality Withmentioning

confidence: 99%

“…Applying the expressions obtained for nominator and denominator of the latter fraction in (70) and taking into account that a function P(X ∈ B(x, t)) is continuous in (x, t) ∈ R d × R + (see, e.g., Lemma 1 in [10]) we get, for each n ∈ N (n > 1), k ∈ {1, . .…”

Section: Appendixmentioning

confidence: 99%

See 3 more Smart Citations

Statistical estimation of conditional Shannon entropy

Bulinski

Kozhevin

2019

ESAIM: PS

Self Cite

View full text Add to dashboard Cite

The new estimates of the conditional Shannon entropy are introduced in the framework of the model describing a discrete response variable depending on a vector of d factors having a density w.r.t. the Lebesgue measure in R d . Namely, the mixed-pair model (X, Y ) is considered where X and Y take values in R d and an arbitrary finite set, respectively. Such models include, for instance, the famous logistic regression. In contrast to the well-known Kozachenko -Leonenko estimates of unconditional entropy the proposed estimates are constructed by means of the certain spacial order statistics (or k-nearest neighbor statistics where k = k n depends on amount of observations n) and a random number of i.i.d. observations contained in the balls of specified random radii. The asymptotic unbiasedness and L 2 -consistency of the new estimates are established under simple conditions. The obtained results can be applied to the feature selection problem which is important, e.g., for medical and biological investigations.

show abstract

Section: Introductionmentioning

confidence: 90%

Section: Introductionmentioning

confidence: 95%

“…Let X ∼ N (a, Σ) where a ∈ R d and Σ > 0. It is easily seen that, for any ε > 0, one has E| log f X (X)| ε < ∞ (see, e.g., [10]). Since, for x ∈ R d and y ∈ M ,…”

Section: Applying the Hoeffding Inequality Withmentioning

confidence: 99%

Section: Appendixmentioning

confidence: 99%

See 2 more Smart Citations

Statistical estimation of conditional Shannon entropy

Bulinski

Kozhevin

2019

ESAIM: PS

Self Cite

View full text Add to dashboard Cite

show abstract

“…There has been quite a large amount of recent work [Nguyen et al, 2010, Liu et al, 2012, Moon and Hero, 2014b, Singh and Póczos, 2014a,b, Krishnamurthy et al, 2014, Moon and Hero, 2014a, Gao et al, 2015a,b, Mukherjee et al, 2015, Moon et al, 2016, Berrett et al, 2016, Gao et al, 2017b, Noshad et al, 2017, Wisler et al, 2017, Singh and Pøczos, 2017, Noshad and Hero III, 2018, Bulinski and Dimitrov, 2018 on practical estimation of nonlinear integral functionals of probability densities, of the form…”

Section: Estimation Of Related Functionalsmentioning

confidence: 99%

Minimax Estimation of Quadratic Fourier Functionals

Singh¹,

Sriperumbudur²,

Póczos³

2018

Preprint

View full text Add to dashboard Cite

We study estimation of (semi-)inner products between two nonparametric probability distributions, given IID samples from each distribution. These products include relatively well-studied classical L 2 and Sobolev inner products, as well as those induced by translation-invariant reproducing kernels, for which we believe our results are the first. We first propose estimators for these quantities, and the induced (semi)norms and (pseudo)metrics. We then prove non-asymptotic upper bounds on their mean squared error, in terms of weights both of the inner product and of the two distributions, in the Fourier basis. Finally, we prove minimax lower bounds that imply rate-optimality of the proposed estimators over Fourier ellipsoids. Singh, Sriperumbudur, and Póczosas well as the squared seminorm P 2 a := P, P a and squared pseudometric P − Q 2 a , using n i.i.d. samples X 1 , ..., X n i.i.d.∼ P and Y 1 , ..., Y n i.i.d.∼ Q from each distribution. Specifically, we assume that P and Q lie in a smaller subspace H b ⊆ H a parameterized by a Z-indexed real family b = {b z } z∈Z . In this setting, we study the minimax L 2 error M (a, b) of estimating P, Q a , over P and Q lying in a (unit) ellipsoid with respect to• b ; that is, the quantity M (a, b

show abstract

Statistical estimation of the Shannon entropy

Cited by 2 publications

References 0 publications

Statistical estimation of conditional Shannon entropy

Statistical estimation of conditional Shannon entropy

Minimax Estimation of Quadratic Fourier Functionals

Contact Info

Product

Resources

About