Multivariate Extension of Matrix-based Renyi's α-order Entropy Functional

Yu, Shujian; Giraldo, Luis G. Sánchez; Jenssen, Robert; Prı́ncipe, José C.

doi:10.1109/tpami.2019.2932976

Cited by 64 publications

(89 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, it is worth noting that, the proposed methodologies can be simply applied to other DNN architectures, e.g., CNN [16]. The only difference is that we need to use the multivariate extension of matrix-based Renyi's entropy functional [78] to quantify the information flow of CNN.…”

Section: Discussionmentioning

confidence: 99%

Understanding autoencoders with information theoretic concepts

Prı́ncipe

2019

Neural Networks

Self Cite

120

View full text Add to dashboard Cite

Despite their great success in practical applications, there is still a lack of theoretical and systematic methods to analyze deep neural networks. In this paper, we illustrate an advanced information theoretic methodology to understand the dynamics of learning and the design of autoencoders, a special type of deep learning architectures that resembles a communication channel. By generalizing the information plane to any cost function, and inspecting the roles and dynamics of different layers using layer-wise information quantities, we emphasize the role that mutual information plays in quantifying learning from data. We further suggest and also experimentally validate, for mean square error training, three fundamental properties regarding the layer-wise flow of information and intrinsic dimensionality of the bottleneck layer, using respectively the data processing inequality and the identification of a bifurcation point in the information plane that is controlled by the given data. Our observations have direct impact on the optimal design of autoencoders, the design of alternative feedforward training methods, and even in the problem of generalization.

show abstract

Section: Discussionmentioning

confidence: 99%

Understanding autoencoders with information theoretic concepts

Prı́ncipe

2019

Neural Networks

Self Cite

120

View full text Add to dashboard Cite

show abstract

“…In this section, we start with a brief introduction to the recently proposed matrix-based Rényi’s

-entropy functional and its multivariate extension [ 13 ]. The novel definition yields two simple stopping criteria as presented below.…”

Section: Simple Stopping Criteria For Information Theoretic Featurmentioning

confidence: 99%

“…Unfortunately, the accurate PDF estimation impedes its more widespread adoption in data driven science. To solve this problem, References [ 13 , 23 ] suggest similar quantities that resemble quantum Rényi’s entropy [ 27 ] in terms of the normalized eigenspectrum of the Hermitian matrix of the projected data in RKHS, thus estimating the entropy and joint entropy among two or multiple variables directly from data without PDF estimation. For brevity, we directly give the definition.…”

Section: Simple Stopping Criteria For Information Theoretic Featurmentioning

confidence: 99%

“…Regarding the first problem, our recent work [ 13 ] suggested that

can be efficiently estimated using the normalized eigenspectrum of a Hermitian matrix of the projected data in the reproducing kernel Hilbert space (RKHS). In this paper, we expand on the topic of Reference [ 13 ] and illustrate that the novel multivariate matrix-based Rényi’s

-entropy functional also enables simple strategies to guide the early stopping in the greedy search procedure of information theoretic feature selection methods.…”

Section: Introductionmentioning

confidence: 99%

“…Admittedly, one can say that we can stop the selection if the increment of

or the decrement of

approaches zero with a tiny residual. Unfortunately, since we still do not have a reliable estimator of MI and CMI in high-dimensional space (before [ 13 ]), it is hard for us to measure or determine how small the residual terms are.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Simple Stopping Criteria for Information Theoretic Feature Selection

Prı́ncipe

2019

Entropy

Self Cite

View full text Add to dashboard Cite

Feature selection aims to select the smallest feature subset that yields the minimum generalization error. In the rich literature in feature selection, information theory-based approaches seek a subset of features such that the mutual information between the selected features and the class labels is maximized. Despite the simplicity of this objective, there still remain several open problems in optimization. These include, for example, the automatic determination of the optimal subset size (i.e., the number of features) or a stopping criterion if the greedy searching strategy is adopted. In this paper, we suggest two stopping criteria by just monitoring the conditional mutual information (CMI) among groups of variables. Using the recently developed multivariate matrix-based Rényi's α-entropy functional, which can be directly estimated from data samples, we showed that the CMI among groups of variables can be easily computed without any decomposition or approximation, hence making our criteria easy to implement and seamlessly integrated into any existing information theoretic feature selection methods with a greedy search strategy. Index TermsFeature selection, stopping criterion, conditional mutual information, Multivariate matrix-based Rényi's α-entropy functional.

show abstract