Variable selection for regression problems using Gaussian mixture models to estimate mutual information

Eirola, Emil; Lendasse, Amaury; Karhunen, Juha

doi:10.1109/ijcnn.2014.6889561

Cited by 5 publications

(4 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The idea of using Mutual Information to carry out feature selection has been proposed in the literature in several ways [ 28 , 29 , 30 , 31 ]. Let

and

for

be the input sample pairs, then, the Mutual Information between these two variables can be defined as the amount of information that can be extracted from Y given X .…”

Section: Experiments and Discussionmentioning

confidence: 99%

A Comparative Analysis of Machine Learning Techniques for Muon Count in UHECR Extensive Air-Showers

Guillén

Martínez

Carceller

et al. 2020

Entropy

View full text Add to dashboard Cite

The main goal of this work is to adapt a Physics problem to the Machine Learning (ML) domain and to compare several techniques to solve it. The problem consists of how to perform muon count from the signal registered by particle detectors which record a mix of electromagnetic and muonic signals. Finding a good solution could be a building block on future experiments. After proposing an approach to solve the problem, the experiments show a performance comparison of some popular ML models using two different hadronic models for the test data. The results show that the problem is suitable to be solved using ML as well as how critical the feature selection stage is regarding precision and model complexity.

show abstract

“…The idea of using Mutual Information to carry out feature selection has been proposed in the literature in several ways [ 28 , 29 , 30 , 31 ]. Let

and

for

be the input sample pairs, then, the Mutual Information between these two variables can be defined as the amount of information that can be extracted from Y given X .…”

Section: Experiments and Discussionmentioning

confidence: 99%

A Comparative Analysis of Machine Learning Techniques for Muon Count in UHECR Extensive Air-Showers

Guillén

Martínez

Carceller

et al. 2020

Entropy

View full text Add to dashboard Cite

show abstract

“…To address these requirements, we present Gaussian mixture model (GMM)-MI (pronounced 'Jimmie'), an algorithm to estimate the full distribution of I(X, Y) based on fitting samples drawn from the distribution with GMMs. While the use of GMMs to estimate MI is not new [81][82][83][84][85][86], these previous works only considered MI in the context of feature selection, and did not carry out uncertainty quantification on the relevant MI estimates, which is critical when using MI to interpret DL models. GMM-MI has been designed to be a robust and flexible tool that can be applied to multiple settings where MI estimation is required.…”

Section: Introductionmentioning

confidence: 99%

A robust estimator of mutual information for deep learning interpretability

Piras¹,

Peiris²,

Pontzen³

et al. 2023

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

We develop the use of mutual information (MI), a well-established metric in information theory, to interpret the inner workings of deep learning models. To accurately estimate MI from a finite number of samples, we present GMM-MI (pronounced “Jimmie”), an algorithm based on Gaussian mixture models that can be applied to both discrete and continuous settings. GMM-MI is computationally efficient, robust to the choice of hyperparameters and provides the uncertainty on the MI estimate due to the finite sample size. We extensively validate GMM-MI on toy data for which the ground truth MI is known, comparing its performance against established mutual information estimators. We then demonstrate the use of our MI estimator in the context of representation learning, working with synthetic data and physical datasets describing highly non-linear processes. We train deep learning models to encode high-dimensional data within a meaningful compressed (latent) representation, and use GMM-MI to quantify both the level of disentanglement between the latent variables, and their association with relevant physical quantities, thus unlocking the interpretability of the latent representation. Upon acceptance of this work, we will link a publicly available GitHub repository which contains GMM-MI and the code to reproduce the results of this paper.

show abstract

“…To address these requirements, we present GMM-MI (pronounced "Jimmie"), an algorithm to estimate the full distribution of I(X, Y ) based on fitting samples drawn from the distribution with Gaussian mixture models (GMMs). While the use of GMMs to estimate MI is not new [81][82][83][84][85][86], these previous works only considered MI in the context of feature selection, and did not carry out uncertainty quantification on the relevant MI estimates. GMM-MI has been designed to be a flexible tool that can be applied to multiple settings where MI estimation is required.…”

Section: Introductionmentioning

confidence: 99%

A robust estimator of mutual information for deep learning interpretability

Piras,

Peiris,

Pontzen

et al. 2022

Preprint

View full text Add to dashboard Cite

We develop the use of mutual information (MI), a well-established metric in information theory, to interpret the inner workings of deep learning models. To accurately estimate MI from a finite number of samples, we present GMM-MI (pronounced "Jimmie"), an algorithm based on Gaussian mixture models that can be applied to both discrete and continuous settings. GMM-MI is computationally efficient, robust to the choice of hyperparameters and provides the uncertainty on the MI estimate due to the finite sample size. We extensively validate GMM-MI on toy data for which the ground truth MI is known, comparing its performance against established mutual information estimators. We then demonstrate the use of our MI estimator in the context of representation learning, working with synthetic data and physical datasets describing highly non-linear processes. We train deep learning models to encode high-dimensional data within a meaningful compressed (latent) representation, and use GMM-MI to quantify both the level of disentanglement between the latent variables, and their association with relevant physical quantities, thus unlocking the interpretability of the latent representation. We make GMM-MI publicly available in this GitHub repository.

show abstract

Variable selection for regression problems using Gaussian mixture models to estimate mutual information

Cited by 5 publications

References 31 publications

A Comparative Analysis of Machine Learning Techniques for Muon Count in UHECR Extensive Air-Showers

A Comparative Analysis of Machine Learning Techniques for Muon Count in UHECR Extensive Air-Showers

A robust estimator of mutual information for deep learning interpretability

A robust estimator of mutual information for deep learning interpretability

Contact Info

Product

Resources

About