Neural Estimator of Information for Time-Series Data with Dependency

Abstract

Novel approaches to estimate information measures using neural networks are well-celebrated in recent years both in the information theory and machine learning communities. These neural-based estimators are shown to converge to the true values when estimating mutual information and conditional mutual information using independent samples. However, if the samples in the dataset are not independent, the consistency of these estimators requires further investigation. This is of particular interest for a more complex measure such as the directed information, which is pivotal in characterizing causality and is meaningful over time-dependent variables. The extension of the convergence proof for such cases is not trivial and demands further assumptions on the data. In this paper, we show that our neural estimator for conditional mutual information is consistent when the dataset is generated with samples of a stationary and ergodic source. {In other words, we show that our information estimator using neural networks converges asymptotically to the true value with probability one. Besides universal functional approximation of neural networks, a core lemma to show the convergence is Birkhoff's ergodic theorem. Additionally, we use the technique to estimate directed information and demonstrate the effectiveness of our approach in simulations.

Neural Estimation and Optimization of Directed Information over Continuous Spaces

“…The latter, in particular, showed that MINE is minimax optimal under appropriate regularity assumptions on the distributions (see also [31] for formal limitations on MINE performance). For data with memory, [32] leveraged MINE for transfer entropy, while [33] constructed a conditional MI estimator and extended it to DI between 1st order Markov processes.…”

LPOT: Locality-Preserving Gromov–Wasserstein Discrepancy for Nonrigid Point Set Registration

“…The mutual information NE (MINE) was proposed in [BBR + 18], and has seen various improvements since [POvdO + 18, SE19, CABH + 19, MMD + 21]. Extensions of the neural estimation approach to directed information were studied in [MGBS21,TAGP23b,TAGP23a]. Theoretical guarantees for f -divergence NEs, accounting for approximation and estimation errors, as we do here, were developed in [SSG21,SG22] (see also [NWJ10] for a related approach based on reproducing kernel Hilbert space parameterization).…”

LogNNet Neural Network Application for Diabetes Mellitus Diagnosis

“…The simplicity of the training method in reservoir networks attracts researchers from related scientific fields. Most of these studies are related to traditional applications of machine learning methods: pattern recognition [7], system approximation [8], adaptive data filtering [9]. To improve the classification accuracy and enhance the approximation capabilities of reservoir networks, some studies employ a combination of multiple reservoirs, which increases the computational resource requirements.…”