On mutual information estimation for mixed-pair random variables

Beknazaryan, Aleksandr; Dang, Xin; Sang, Hailin

doi:10.1016/j.spl.2018.12.011

Cited by 9 publications

(6 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mutual information is a good measure of a nonlinear relationship. , In probability theory, the mutual information measures the mutual dependence between the two variables, which indicates how much information is communicated in one random variable about another. For two random variables x and y with the given n samples { x 1 , x 2 , ···, x n } and { y 1 , y 2 , ···, y n }, the mutual information between x and y is defined as where p ( x i ) and p ( y j ) are the marginal probability density function of x and y , respectively, while p ( x i , y i ) is the joint probability density function of x and y .…”

Section: The Proposed Tslkpca Methodsmentioning

confidence: 99%

Two-Step Localized Kernel Principal Component Analysis Based Incipient Fault Diagnosis for Nonlinear Industrial Processes

Deng

Cai

Cao

et al. 2020

Ind. Eng. Chem. Res.

View full text Add to dashboard Cite

Kernel principal component analysis (KPCA) has been widely applied to the nonlinear process fault diagnosis field. However, it often does not perform well in the case of incipient faults because of the omission of local data information. To overcome this problem, one enhanced KPCA method, called the two-step localized KPCA (TSLKPCA), is proposed for incipient fault diagnosis in this work. The two steps are designed to mine the local data information better. At the first step, the KPCA optimization objective is modified by integrating the local structure preservation so that the extracted kernel components preserve the global and local data structure information simultaneously. At the second step, for the extracted kernel components, the local probability information is further mined by the Kullback Leibler divergence (KLD), which measures the variations of the kernel components’ probability distributions. On the basis of these two steps, the original kernel components are transformed into the KLD components, and the corresponding model is developed for incipient fault detection. To isolate the faulty variables, the contribution plot is constructed based on the mutual information between the measured variables and the KLD components obtained by TSLKPCA. Finally, two simulations of a numerical example and the continuous stirred tank reactor (CSTR) control system show that the proposed method has good incipient fault detection and diagnostic performance.

show abstract

Section: The Proposed Tslkpca Methodsmentioning

confidence: 99%

Two-Step Localized Kernel Principal Component Analysis Based Incipient Fault Diagnosis for Nonlinear Industrial Processes

Deng

Cai

Cao

et al. 2020

Ind. Eng. Chem. Res.

View full text Add to dashboard Cite

show abstract

“…which many estimation methods including our own, rely on. However, Beknazaryan et al (2019) give a simple sufficient condition for the decomposition in Eq. ( 2) to hold, which they term the good mixed-pair assumption.…”

Section: Estimation Challengesmentioning

confidence: 99%

“…Another challenge for estimating mutual information in our case is due to the nature of W. Several mixed-pair estimation methods, including Beknazaryan et al (2019), may not apply to our case because they require estimating H(P t | W = w) for each given value of W separately; see App. A for more details.…”

Section: Estimation Challengesmentioning

confidence: 99%

Quantifying the redundancy between prosody and text

Wolf,

Pimentel,

Fedorenko

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Prosody-the suprasegmental component of speech, including pitch, loudness, and tempocarries critical aspects of meaning. However, the relationship between the information conveyed by prosody vs. by the words themselves remains poorly understood. We use large language models (LLMs) to estimate how much information is redundant between prosody and the words themselves. Using a large spoken corpus of English audiobooks, we extract prosodic features aligned to individual words and test how well they can be predicted from LLM embeddings, compared to non-contextual word embeddings. We find a high degree of redundancy between the information carried by the words and prosodic information across several prosodic features, including intensity, duration, pauses, and pitch contours. Furthermore, a word's prosodic information is redundant with both the word itself and the context preceding as well as following it. Still, we observe that prosodic features can not be fully predicted from text, suggesting that prosody carries information above and beyond the words. Along with this paper, we release a general-purpose data processing pipeline for quantifying the relationship between linguistic information and extra-linguistic features. https://github.com/lu-wo/ quantifying-redundancy

show abstract

“…A current limitation of the algorithm is that it is limited to homogeneous data sets and can not be applied to mixed data types. Even though though initial proposals for mixed-type MI estimators exist (e.g., Ross (2014); Gao et al (2017); Rahimzamani et al (2018); Beknazaryan et al (2019)), their integration into the proposed forward-selection algorithm is not straightforward and is subject to future work.…”

Section: Choice Of Estimators For CMI Estimationmentioning

confidence: 99%

A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition

Schmitt¹,

Wibral²

2021

Preprint

View full text Add to dashboard Cite

Selecting a minimal feature set that is maximally informative about a target variable is a central task in machine learning and statistics. Information theory provides a powerful framework for formulating feature selection algorithms-yet, a rigorous, information-theoretic definition of feature relevancy, which accounts for feature interactions such as redundant and synergistic contributions, is still missing. We argue that this lack is inherent to classical information theory which does not provide measures to decompose the information a set of variables provides about a target into unique, redundant, and synergistic contributions. Such a decomposition has been introduced only recently by the partial information decomposition (PID) framework. Using PID, we clarify why feature selection is a conceptually difficult problem when approached using information theory and provide a novel definition of feature relevancy and redundancy in PID terms. From this definition, we show that the conditional mutual information (CMI) maximizes relevancy while minimizing redundancy and propose an iterative, CMI-based algorithm for practical feature selection. We demonstrate the power of our CMI-based algorithm in comparison to the unconditional mutual information on benchmark examples and provide corresponding PID estimates to highlight how PID allows to quantify information contribution of features and their interactions in feature-selection problems.

show abstract

On mutual information estimation for mixed-pair random variables

Cited by 9 publications

References 11 publications

Two-Step Localized Kernel Principal Component Analysis Based Incipient Fault Diagnosis for Nonlinear Industrial Processes

Two-Step Localized Kernel Principal Component Analysis Based Incipient Fault Diagnosis for Nonlinear Industrial Processes

Quantifying the redundancy between prosody and text

A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition

Contact Info

Product

Resources

About