Information-theoretic model selection for optimal prediction of stochastic dynamical systems from data

Darmon, David

doi:10.1103/physreve.97.032206

Cited by 7 publications

(4 citation statements)

References 73 publications

(94 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…wheref −t:l is the conditional density with the points removed [54]. A suggested approach is to take l = 0 and only remove the individual observation [96]. In practise, it is advised to fix p and then calculate the bandwidths due to the computational complexity of the cross-validation [54].…”

Section: Specific Entropymentioning

confidence: 99%

A Review of Shannon and Differential Entropy Rate Estimation

Feutrill

Roughan

2021

Entropy

View full text Add to dashboard Cite

In this paper, we present a review of Shannon and differential entropy rate estimation techniques. Entropy rate, which measures the average information gain from a stochastic process, is a measure of uncertainty and complexity of a stochastic process. We discuss the estimation of entropy rate from empirical data, and review both parametric and non-parametric techniques. We look at many different assumptions on properties of the processes for parametric processes, in particular focussing on Markov and Gaussian assumptions. Non-parametric estimation relies on limit theorems which involve the entropy rate from observations, and to discuss these, we introduce some theory and the practical implementations of estimators of this type.

show abstract

Section: Specific Entropymentioning

confidence: 99%

A Review of Shannon and Differential Entropy Rate Estimation

Feutrill

Roughan

2021

Entropy

View full text Add to dashboard Cite

show abstract

“…The suggested technique for selecting p is a cross-validation technique which removes an individual observation and l observations either side. A suggested approach is to take l = 0 and only remove the individual observation [9]. In practice, it is advised to fix a p and then calculate the bandwidths due to the computational complexity of the cross-validation [8].…”

Section: Specific Entropymentioning

confidence: 99%

NPD Entropy: A Non-Parametric Differential Entropy Rate Estimator

Feutrill,

Roughan

2021

Preprint

View full text Add to dashboard Cite

The estimation of entropy rates for stationary discrete-valued stochastic processes is a well studied problem in information theory. However, estimating the entropy rate for stationary continuous-valued stochastic processes has not received as much attention. In fact, many current techniques are not able to accurately estimate or characterise the complexity of the differential entropy rate for strongly correlated processes, such as Fractional Gaussian Noise and ARFIMA(0,d,0). To the point that some cannot even detect the trend of the entropy rate, e.g., when it increases/decreases, maximum, or asymptotic trends, as a function of their Hurst parameter. However, a recently developed technique provides accurate estimates at a high computational cost. In this paper, we define a robust technique for non-parametrically estimating the differential entropy rate of a continuous valued stochastic process from observed data, by making an explicit link between the differential entropy rate and the Shannon entropy rate of a quantised version of the original data. Estimation is performed by a Shannon entropy rate estimator, and then converted to a differential entropy rate estimate. We show that this technique inherits many important statistical properties from the Shannon entropy rate estimator. The estimator is able to provide better estimates than the defined relative measures and much quicker estimates than known absolute measures, for strongly correlated processes. Finally, we analyse the complexity of the estimation technique and test the robustness to non-stationarity, and show that none of the current techniques are robust to non-stationarity, even if they are robust to strong correlations.

show abstract

“…We seek the model order that results in a minimum uncertainty. Using the information theoretic criterion from [ 31 ], the model order is chosen to minimize the negative log predictive likelihood (NLPL)

where N is the number of points in the time series, and

is an estimator of the predictive density estimated holding out the block

. We use a kernel-nearest neighbor estimator for f , which performs kernel density estimation over the set of nearest neighbors in the future space [ 32 , 33 ].…”

Section: Definition and Estimation Of Local And Specific Entropy Rmentioning

confidence: 99%

Information Dynamics of a Nonlinear Stochastic Nanopore System

Gilpin

Darmon

Siwy

et al. 2018

Entropy

Self Cite

View full text Add to dashboard Cite

Nanopores have become a subject of interest in the scientific community due to their potential uses in nanometer-scale laboratory and research applications, including infectious disease diagnostics and DNA sequencing. Additionally, they display behavioral similarity to molecular and cellular scale physiological processes. Recent advances in information theory have made it possible to probe the information dynamics of nonlinear stochastic dynamical systems, such as autonomously fluctuating nanopore systems, which has enhanced our understanding of the physical systems they model. We present the results of local (LER) and specific entropy rate (SER) computations from a simulation study of an autonomously fluctuating nanopore system. We learn that both metrics show increases that correspond to fluctuations in the nanopore current, indicating fundamental changes in information generation surrounding these fluctuations.of as the rate of information generation at a given time point. The specific entropy rate (SER) represents the statistical uncertainty in an as yet unobserved future, given a specific past [24]. The LER, in isolation, is a retrospective measure, and the SER is a prospective measure, and each yields distinct information about the behavior of the dynamical system.In [25], the authors use current-voltage data collected from a single, conical nanopore in contact with an ion solution to motivate a mechanistic model for the electronic behavior of the nanopore system. The pore is externally biased, but its fluctuations (openings and closings) are autonomous and result from dynamic formation and dissolution (or passage) of tiny inorganic precipitates [26,27]. They demonstrated that their mathematical model captures important properties of the observable from the nanopore system, including these fluctuations, which appear as rapid transitions between high and low conductance states in the time series. One reason this system is an ideal candidate for analysis via information dynamics is that its fluctuations may be informed by its previous states (transitions may depend on previous states, as they do in a Markov process).The structure of this paper is as follows. In Section 2, we present the definitions and estimation procedures for local and specific entropy rates. We introduce the model of the nanopore in Section 3. We then investigate the properties of the nanopore system as viewed through the lens of information dynamics in Section 4. Finally, in Section 5, we conclude by considering other potential avenues of study for this and other nanoscale systems.

show abstract

Information-theoretic model selection for optimal prediction of stochastic dynamical systems from data

Cited by 7 publications

References 73 publications

A Review of Shannon and Differential Entropy Rate Estimation

A Review of Shannon and Differential Entropy Rate Estimation

NPD Entropy: A Non-Parametric Differential Entropy Rate Estimator

Information Dynamics of a Nonlinear Stochastic Nanopore System

Contact Info

Product

Resources

About