Nearest neighbor estimate of conditional mutual information in feature selection

Tsimpiris, Alkiviadis; Vlachos, Ioannis; Kugiumtzis, Dimitris

doi:10.1016/j.eswa.2012.05.014

Cited by 33 publications

(34 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, unlike the discrete case, computing CMI for continuous variables requires estimating probability density functions (pdf) from the available data samples, which is not a straightforward task, and in fact, considered as one of the main problems associated with applying this measure. With this regard, different possible non-parametric approaches exist in the literature, of which we select the k-nearest neighbour estimator [9,14]. Unlike many other approaches, this estimator allows accommodating high variable dimensionality (the estimator of the CMI for context variables must be able to work well with higher dimensions) and produces more accurate results.…”

Section: Computational Solutionmentioning

confidence: 98%

Context Identification and Exploitation in Dynamic Data Mining - An Application to Classifying Electricity Price Changes

Barakat

2014

Adaptive and Intelligent Systems

View full text Add to dashboard Cite

Abstract. Adapting classification models to concept changes is one of the main challenges associated with learning in dynamic environments, where the definition of the target concept may change over time under the influence of varying contextual factors. Existing adaptive approaches, however, are limited in terms of the extent to which such contextual factors are explicitly identified and utilised, despite their importance. In response, we propose an informationtheoretic-based approach for systematic context identification, aiming to learn from data the contextual characteristics of the domain by identifying the context variables contributing to concept changes. Such explicit identification of context enables capturing the causes of drift, and hence facilitating more effective adaptation. We conduct experimental analyses to demonstrate the effectiveness of the approach on both simulated datasets with various change scenarios, and on an actual benchmark dataset from an electricity market.

show abstract

Section: Computational Solutionmentioning

confidence: 98%

Context Identification and Exploitation in Dynamic Data Mining - An Application to Classifying Electricity Price Changes

Barakat

2014

Adaptive and Intelligent Systems

View full text Add to dashboard Cite

show abstract

“…It should be noted that the above integrals may result in a heavy computation burden. Therefore, the above marginal entropies and joint entropy are transferred into a simpler expression using the k‐nearest neighbours algorithm (kNN) and the Kozachenko‐Leonenko estimator for Shannon entropies . According to the concept of nearest neighbour, Equation can be expressed as:

H true(z, v true) = \frac{d i s_{z} + d i s_{v}}{N} \sum_{i = 1}^{N} ϵ (i) + ψ (N) - ψ true(k true) + \log true(c_{d i {s z}_{}} c_{d i {s v}_{}} true)

where

ϵ_{z} (i) / 2

and

ϵ_{v} (i) / 2

are the distances between the same points projected into the z and v subspaces, respectively.…”

Section: Preliminariesmentioning

confidence: 99%

“…Based on the DMIS strategy, the moving windows with similar dynamics are assigned into a cluster. Mutual information is a Shannon entropy based technique that can measure the information shared by two datasets with different dimensions . Compared to PCA based similarity measurement methods, the mutual information method does not need any established model or iterative calculation.…”

Section: Introductionmentioning

confidence: 99%

Dynamic mutual information similarity based transient process identification and fault detection

Yuchen

Zhou

et al. 2018

Can J Chem Eng

View full text Add to dashboard Cite

Industrial process status can be modified according to continuous operations, which are required by different production specifications. Commonly, in a multimode process, attention was always paid to stable modes, while transitions were neglected. In a transition process, the process may be externally time varying or nonstationary so that the identification and the modelling are intractable to implement using traditional statistical methods. In this article, a transition identification and process monitoring method is proposed to handle the above problems based on a novel dynamic mutual information similarity (DMIS) analysis. Firstly, a multimode process is represented by a series of overlapping moving windows to consider the local information. In order to extract the corresponding dynamic information, each of these windows is modelled using the dynamic partial least squares (DPLS) method. Then, the mutual information algorithm is introduced to calculate the similarity between different latent variables. The hierarchical clustering method is employed to transfer the similarity information into a visualized dendrogram where the whole process, including a transition, is divided into several segments. The statistical characteristics in each segment are relatively stable and can be characterized by conventional multivariate statistical process control methods. Online identification and fault detection are then carried out for the multimode process through a series of DPLS models, established in the offline steps. The feasibility and effectiveness of the proposed method are validated by the Tennessee Eastman (TE) benchmark and a real process. The results of the proposed methods have shown superior performance compared to previous works.

show abstract

“…It is worth noting that the estimation of mutual information through computing the integrals and summations is intensive and inefficient in Equation (17) in practice. In order to reduce the computation burden, the nearest neighbour strategy which based on the Kozachenko–Leonenko estimator of Shannon entropy has been proposed to work well for estimating the mutual information numerically . Firstly, the estimate of joint entropy through nearest neighbour technique is

H (x, y) = - ψ (l) + ψ (n) + \log (c_{d_{x}} c_{d_{y}}) + \frac{d_{x} + d_{y}}{n} \sum_{i = 1}^{n} ε false(i false)

where

ε (i) = \max {ε_{x} (i), ε_{y} (i)}

is the maximum Euclidean norm of the

i

th sample point…”

Section: Variable Moving Windows Based Non‐gaussian Dissimilarity Anamentioning

confidence: 99%

Variable moving windows based non‐Gaussian dissimilarity analysis technique for batch processes fault detection and diagnosis

Zhang

2015

Can J Chem Eng

View full text Add to dashboard Cite

In this paper, a novel variable moving windows based non‐Gaussian dissimilarity analysis technique is developed to handle the challenges related to the batch processes such as non‐linearity, non‐Gaussianity and time‐varying dynamics. First, the independent component analysis (ICA) models are developed on the normal reference data sets and the monitored data sets to extract the dominant independent component subspaces through a variable moving windows strategy. Then, non‐Gaussian dissimilarity indices are computed to evaluate the statistical independency of the extracted IC subspaces at each specific time interval. Thus, the process non‐Gaussian features are fully captured and the process trajectories distribution information of batch‐to‐batch is quantitatively estimated for online process monitoring. Further, the non‐Gaussian contribution index based on the mutual information is introduced to identify the variables that may be responsible for the process abnormality. The reliability and validity of the proposed method are verified on the fed‐batch Penicillin Fermentation process. The application results present superior fault detection and diagnosis performance compared with the PCA dissimilarity, MPCA and MICA approaches.

show abstract

Nearest neighbor estimate of conditional mutual information in feature selection

Cited by 33 publications

References 32 publications

Context Identification and Exploitation in Dynamic Data Mining - An Application to Classifying Electricity Price Changes

Context Identification and Exploitation in Dynamic Data Mining - An Application to Classifying Electricity Price Changes

Dynamic mutual information similarity based transient process identification and fault detection

Variable moving windows based non‐Gaussian dissimilarity analysis technique for batch processes fault detection and diagnosis

Contact Info

Product

Resources

About