k-nearest neighbor estimation of entropies with confidence

Sricharan, Kumar; Raich, Raviv; Hero, Alfred O.

doi:10.1109/isit.2011.6033726

Cited by 11 publications

(8 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are many different approaches for estimating information measures, including kernel based methods, nearest neighbor methods, methods based on sample distances as well as multiple variants of plug-in estimates. Many estimators have been shown to be consistent and/or asymptotically unbiased under various constraints, e. g., in [17,1,12,21,36]. An excellent overview can be found in [6].…”

Section: Previous Workmentioning

confidence: 99%

On the Estimation of Information Measures of Continuous Distributions

Pichler¹,

Piantanida²,

Koliander³

2020

Preprint

View full text Add to dashboard Cite

The estimation of information measures of continuous distributions based on samples is a fundamental problem in statistics and machine learning. In this paper, we analyze estimates of differential entropy in K-dimensional Euclidean space, computed from a finite number of samples, when the probability density function belongs to a predetermined convex family P. First, estimating differential entropy to any accuracy is shown to be infeasible if the differential entropy of densities in P is unbounded, clearly showing the necessity of additional assumptions. Subsequently, we investigate sufficient conditions that enable confidence bounds for the estimation of differential entropy. In particular, we provide confidence bounds for simple histogram based estimation of differential entropy from a fixed number of samples, assuming that the probability density function is Lipschitz continuous with known Lipschitz constant and known, bounded support. Our focus is on differential entropy, but we provide examples that show that similar results hold for mutual information and relative entropy as well.

show abstract

Section: Previous Workmentioning

confidence: 99%

On the Estimation of Information Measures of Continuous Distributions

Pichler¹,

Piantanida²,

Koliander³

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Seo et al in [41] proposed a data anomaly detection method for micro-clustering, and designed a method for detecting and specifying outliers using a local outlier as a center of a micro-cluster in offline components. Sricharan et al in [42] defined the outliers of data in WSN, and proposed a method based on data classification to estimate and calculate by probability density function. This method has been proven to be applicable to different types of data testing, including Gaussian distribution.…”

Section: Related Workmentioning

confidence: 99%

Hierarchical Edge Computing: A Novel Multi-Source Multi-Dimensional Data Anomaly Detection Scheme for Industrial Internet of Things

Peng

Tan

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Every year, many people around the world die because of mining accidents. Industrial Internet of Things (IIoT) can be employed to sense public safety hazards and provide early warning of accidents, thereby ensuring safe operations at underground mining, personnel positioning, and specific items supervision and emergency response. Real-time data anomaly detection can predict the probability of occurrence of the abnormal event. However, massive heterogeneous monitoring data, poor wireless environment and data spatio-temporal association have posed a serious challenge to data anomaly detection for underground mining. Existing methods are mostly concerned about single data or processing at cloud platform, with little regard for the time and space association. Focus on the accuracy and timeliness of data anomaly detection, a novel multi-source multi-dimensional data anomaly detection scheme based on hierarchical edge computing model is presented in this paper. Firstly, a hierarchical edge computing model is proposed to realize load balance and low-latency data processing at the sensor end and base-station end. Then a single-source data anomaly detection algorithm is designed based on fuzzy theory, which can comprehensively analyze the anomaly detection results of multiple consecutive moments. Finally, a multisource data anomaly detection algorithm executed at the base-station end is designed to consider the sensing data associated attributes of time and space. Experimental results reveal that the proposed scheme has higher detection accuracy and lower processing delay compared with traditional solutions. INDEX TERMS Industrial Internet of Things (IIoT), underground mining, anomaly detection, multi-source multi-dimensional data, edge computing.

show abstract

“…In that case, the empirical relative frequencies in these 100 bins are no longer reasonable estimates of the true joint probabilities. How to estimate the entropy under these circumstances has been studied intensely in recent years (Paninski, 2003; Nemenman et al, 2004; Paninski, 2004; Shwartz et al, 2005; Ho et al, 2010; Sricharan et al, 2011). …”

Section: An Information-theoretic Definition Of Temporal Contingencymentioning

confidence: 99%

Temporal contingency

Gallistel

Craig

Shahan

2014

Behavioural Processes

View full text Add to dashboard Cite

Contingency, and more particularly temporal contingency, has often figured in thinking about the nature of learning. However, it has never been formally defined in such a way as to make it a measure that can be applied to most animal learning protocols. We use elementary information theory to define contingency in such a way as to make it a measurable property of almost any conditioning protocol. We discuss how making it a measurable construct enables the exploration of the role of different contingencies in the acquisition and performance of classically and operantly conditioned behavior.

show abstract

k-nearest neighbor estimation of entropies with confidence

Cited by 11 publications

References 21 publications

On the Estimation of Information Measures of Continuous Distributions

On the Estimation of Information Measures of Continuous Distributions

Hierarchical Edge Computing: A Novel Multi-Source Multi-Dimensional Data Anomaly Detection Scheme for Industrial Internet of Things

Temporal contingency

Contact Info

Product

Resources

About