Estimating Mixture Entropy with Pairwise Distances

Kolchinsky, Artemy; Tracey, Brendan D.

doi:10.3390/e19070361

Cited by 97 publications

(112 citation statements)

References 32 publications

Supporting

Mentioning

112

Contrasting

Order By: Relevance

“…Moreover, one should note that although KNN based estimator or KDE is more data efficient, the adaptive bin size will jeopardize the shape of PDF, thus makes the estimator deviates from true mutual information values. In fact, the base estimator used in [36] just provides KDE-based lower and upper bounds on the true mutual information [95]. Interestingly, a recent paper by Noshad et al [93] suggested using dependence graphs to estimate true mutual information values and observed the compression phase even using ReLU activation functions.…”

Section: ) a Deeper Insight On The Role Of Information Theoretic Estmentioning

confidence: 99%

Understanding autoencoders with information theoretic concepts

Prı́ncipe

2019

Neural Networks

120

View full text Add to dashboard Cite

Despite their great success in practical applications, there is still a lack of theoretical and systematic methods to analyze deep neural networks. In this paper, we illustrate an advanced information theoretic methodology to understand the dynamics of learning and the design of autoencoders, a special type of deep learning architectures that resembles a communication channel. By generalizing the information plane to any cost function, and inspecting the roles and dynamics of different layers using layer-wise information quantities, we emphasize the role that mutual information plays in quantifying learning from data. We further suggest and also experimentally validate, for mean square error training, three fundamental properties regarding the layer-wise flow of information and intrinsic dimensionality of the bottleneck layer, using respectively the data processing inequality and the identification of a bifurcation point in the information plane that is controlled by the given data. Our observations have direct impact on the optimal design of autoencoders, the design of alternative feedforward training methods, and even in the problem of generalization.

show abstract

Section: ) a Deeper Insight On The Role Of Information Theoretic Estmentioning

confidence: 99%

Understanding autoencoders with information theoretic concepts

Prı́ncipe

2019

Neural Networks

120

View full text Add to dashboard Cite

show abstract

“…The mutual information involves the entropy of mixture distribution with intractable analytical form. Thus, pairwise-distances are adopted to provide lower bound and upper bound on the mutual information [41]. The results are shown in the following proposition for completeness.…”

Section: Durationmentioning

confidence: 99%

Achievable Rate Bounds on Poisson Channel with a Sample-Based Practical Photon-Counting Receiver

Jiang

Gong

Wang

et al. 2019

2019 IEEE International Conference on Communications Workshops (ICC Workshops)

View full text Add to dashboard Cite

We investigate the achievable rate and capacity of a non-perfect photon-counting receiver. For the case of long symbol duration, the achievable rate under on-off keying modulation is investigated based on Kullback-Leibler (KL) divergence and Chernoff α-divergence. We prove the tightness of the derived bounds for large peak power with zero background radiation with exponential convergence rate, and for low peak power of order two convergence rate. For large peak power with fixed background radiation and low background radiation with fixed peak power, the proposed bound gap is a small positive value for low background radiation and large peak power, respectively. Moreover, we propose an approximation on the achievable rate in the low background radiation and long symbol duration regime, which is more accurate compared with the derived upper and lower bounds in the medium signal to noise ratio (SNR) regime. For the symbol duration that can be sufficiently small, the capacity and the optimal duty cycle are is investigated. We show that the capacity approaches that of continuous Poisson capacity as T s = τ → 0. The asymptotic capacity is analyzed for low and large peak power. Compared with the continuous Poisson capacity, the capacity of a non-perfect receiver is almost lossless and loss with attenuation for low peak power given zero background radiation and nonzero background radiation, respectively. For large peak power, the capacity with a non-perfect receiver converges, while that of continuous Poisson capacity channel linearly increases. The above theoretical results are extensively validated by numerical results.

show abstract

“…Maximization of information or, equivalently, maximization of the output entropy has been proposed by many authors (see, e.g., [13,15,17,18,[21][22][23]), but the mutual information is very hard to compute and the problem is often intractable. To overcome this serious difficulty, instead of mutual information the lower bound, given by Kolchinsky & Tracey [24], has been used. This is a pairwise-distance based entropy estimator and it it useful here, since it is differentiable, tight, and asymptotically reaches the maximum possible information (see [24]…”

Section: Introductionmentioning

confidence: 99%

“…To overcome this serious difficulty, instead of mutual information the lower bound, given by Kolchinsky & Tracey [24], has been used. This is a pairwise-distance based entropy estimator and it it useful here, since it is differentiable, tight, and asymptotically reaches the maximum possible information (see [24]…”

Section: Introductionmentioning

confidence: 99%

Bayesian Input Design for Linear Dynamical Model Discrimination

Bania

2019

Entropy

View full text Add to dashboard Cite

A Bayesian design of the input signal for linear dynamical model discrimination has been proposed. The discrimination task is formulated as an estimation problem, where the estimated parameter indexes particular models. As the mutual information between the parameter and model output is difficult to calculate, its lower bound has been used as a utility function. The lower bound is then maximized under the signal energy constraint. Selection between two models and the small energy limit are analyzed first. The solution of these tasks is given by the eigenvector of a certain Hermitian matrix. Next, the large energy limit is discussed. It is proved that almost all (in the sense of the Lebesgue measure) high energy signals generate the maximum available information, provided that the impulse responses of the models are different. The first illustrative example shows that the optimal signal can significantly reduce error probability, compared to the commonly-used step or square signals. In the second example, Bayesian design is compared with classical average D-optimal design. It is shown that the Bayesian design is superior to D-optimal design, at least in this example. Some extensions of the method beyond linear and Gaussian models are briefly discussed.

show abstract

Estimating Mixture Entropy with Pairwise Distances

Cited by 97 publications

References 32 publications

Understanding autoencoders with information theoretic concepts

Understanding autoencoders with information theoretic concepts

Achievable Rate Bounds on Poisson Channel with a Sample-Based Practical Photon-Counting Receiver

Bayesian Input Design for Linear Dynamical Model Discrimination

Contact Info

Product

Resources

About