Infinite Excess Entropy Processes with Countable-State Generators

Travers, Nicholas F.; Crutchfield, James P.

doi:10.3390/e16031396

Cited by 30 publications

(41 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This implies rather small spectral gaps if the process has a countable infinity of causal states-e.g., as in Ref. [41]-or a distribution of eigenvalues heavily weighted near λ = 0, if the process has an uncountable infinity of causal states. In short, complex processes [4,14] are those for which sequence-based algorithms are most likely to fail.…”

Section: Curse Of Dimensionality In Predictive Rate-distortionmentioning

confidence: 99%

Predictive Rate-Distortion for Infinite-Order Markov Processes

Marzen¹,

Crutchfield

2016

J Stat Phys

View full text Add to dashboard Cite

Predictive rate-distortion analysis suffers from the curse of dimensionality: clustering arbitrarily long pasts to retain information about arbitrarily long futures requires resources that typically grow exponentially with length. The challenge is compounded for infinite-order Markov processes, since conditioning on finite sequences cannot capture all of their past dependencies. Spectral arguments confirm a popular intuition: algorithms that cluster finite-length sequences fail dramatically when the underlying process has long-range temporal correlations and can fail even for processes generated by finite-memory hidden Markov models. We circumvent the curse of dimensionality in rate-distortion analysis of finite-and infinite-order processes by casting predictive rate-distortion objective functions in terms of the forward-and reverse-time causal states of computational mechanics. Examples demonstrate that the resulting algorithms yield substantial improvements.

show abstract

Section: Curse Of Dimensionality In Predictive Rate-distortionmentioning

confidence: 99%

Predictive Rate-Distortion for Infinite-Order Markov Processes

Marzen¹,

Crutchfield

2016

J Stat Phys

View full text Add to dashboard Cite

show abstract

“…An interesting contrast is the log excess entropy divergence seen at the onset of chaos through period-doubling, associated with pushdown stack mechanism [21,Sec. 4.5.1], and seen in the branching copy process [24]. At this stage, though, the possibility of unique associations between the form of information measure divergence and mechanism is not sufficiently well explored.…”

Section: Discussionmentioning

confidence: 99%

“…We calculate E( ) for renewal processes with infinite µ via an analysis technique inspired by Ref. [24] and then calculate E as a limit of E( ) as tends to infinity, seemingly valid for ergodic processes. First, we would like to directly calculate E( ) in terms of forward and reversetime causal states [47]:…”

Section: Appendix B: Renewal Processes With Infinite Mean Intervent Imentioning

confidence: 99%

Statistical signatures of structural organization: The case of long memory in renewal processes

Marzen¹,

Crutchfield²

2016

Physics Letters A

View full text Add to dashboard Cite

Identifying and quantifying memory are often critical steps in developing a mechanistic understanding of stochastic processes. These are particularly challenging and necessary when exploring processes that exhibit long-range correlations. The most common signatures employed rely on second-order temporal statistics and lead, for example, to identifying long memory in processes with power-law autocorrelation function and Hurst exponent greater than 1/2. However, most stochastic processes hide their memory in higher-order temporal correlations. Information measuresspecifically, divergences in the mutual information between a process' past and future (excess entropy) and minimal predictive memory stored in a process' causal states (statistical complexity)-provide a different way to identify long memory in processes with higher-order temporal correlations. However, there are no ergodic stationary processes with infinite excess entropy for which information measures have been compared to autocorrelation functions and Hurst exponents. Here, we show that fractal renewal processes-those with interevent distribution tails ∝ t −α -exhibit long memory via a phase transition at α = 1. Excess entropy diverges only there and statistical complexity diverges there and for all α < 1. When these processes do have power-law autocorrelation function and Hurst exponent greater than 1/2, they do not have divergent excess entropy. This analysis breaks the intuitive association between these different quantifications of memory. We hope that the methods used here, based on causal states, provide some guide as to how to construct and analyze other long memory processes.

show abstract

“…In these examples, the component-process excess entropy I[X −M :0 ; X 0:N |Θ] = I[X −M :0 ; X 0:N |Θ = θ] θ does not diverge with M or N , since finite HMMs have finite excess entropy, which is bounded by the internal state entropy [4,32]. In fact, the excess entropy for many ergodic stochastic processes is finite, even if generated by infinite-state HMMs.…”

Section: Information Analysismentioning

confidence: 99%

“…One of the few tractable and general con- * chaos@ucdavis.edu † smarzen@berkeley.edu structions is the class of Bandit processes constructed from repeated trials of an experiment whose properties are, themselves, varying stochastically from trial to trial [11,12]. Even if each individual trial is a realization generated by a stationary process with finite memory and exponentially decaying correlations, the resulting process over many trials can be infinitary [3][4][5]. Why can the past-future mutual information of Bandit processes diverge?…”

Section: Introductionmentioning

confidence: 99%

Signatures of infinity: Nonergodicity and resource scaling in prediction, complexity, and learning

Crutchfield

Marzen²

2015

Phys. Rev. E

View full text Add to dashboard Cite

We introduce a simple analysis of the structural complexity of infinite-memory processes built from random samples of stationary, ergodic finite-memory component processes. Such processes are familiar from the well known multiarm Bandit problem. We contrast our analysis with computation-theoretic and statistical inference approaches to understanding their complexity. The result is an alternative view of the relationship between predictability, complexity, and learning that highlights the distinct ways in which informational and correlational divergences arise in complex ergodic and nonergodic processes. We draw out consequences for the resource divergences that delineate the structural hierarchy of ergodic processes and for processes that are themselves hierarchical.

show abstract

Infinite Excess Entropy Processes with Countable-State Generators

Abstract: We present two examples of finite-alphabet, infinite excess entropy processes generated by stationary hidden Markov models (HMMs) with countable state sets. The first, simpler example is not ergodic, but the second is. These are the first explicit constructions of processes of this type.

Cited by 30 publications

References 16 publications

Predictive Rate-Distortion for Infinite-Order Markov Processes

Predictive Rate-Distortion for Infinite-Order Markov Processes

Statistical signatures of structural organization: The case of long memory in renewal processes

Signatures of infinity: Nonergodicity and resource scaling in prediction, complexity, and learning

Contact Info

Product

Resources

About