Exact complexity: The spectral decomposition of intrinsic computation

Abstract

We give exact formulae for a wide family of complexity measures that capture the organization of hidden nonlinear processes. The spectral decomposition of operator-valued functions leads to closedform expressions involving the full eigenvalue spectrum of the mixed-state presentation of a process's -machine causal-state dynamic. Measures include correlation functions, power spectra, past-future mutual information, transient and synchronization informations, and many others. As a result, a direct and complete analysis of intrinsic computation is now available for the temporal organization of finitary hidden Markov models and nonlinear dynamical systems with generating partitions and for the spatial organization in one-dimensional systems, including spin systems, cellular automata, and complex materials via chaotic crystallography.© 2016 Elsevier B.V. All rights reserved.The emergence of organization in physical, engineered, and social systems is a fascinating and now, after half a century of active research, widely appreciated phenomenon [1][2][3][4][5]. Success in extending the long list of instances of emergent organization, however, is not equivalent to understanding what organization itself is. How do we say objectively that new organization has appeared? How do we measure quantitatively how organized a system has become?Computational mechanics' answer to these questions is that a system's organization is captured in how it stores and processes information-how it computes [6]. Intrinsic computation was introduced two decades ago to analyze the inherent information processing in complex systems [7]: How much history does a system remember? In what architecture is that information stored? And, how does the system use it to generate future behavior?Computational mechanics, though, is part of a long historical trajectory focused on developing a physics of information [8][9][10]. That nonlinear systems actively process information goes back to Kolmogorov [11], who adapted Shannon's communication theory [12] to measure the information production rate of chaotic dynamical systems. In this spirit, today computational mechanics is routinely used to determine physical and intrinsic computational properties in single-molecule dynamics [13], in complex materials * Corresponding author.E-mail addresses: chaos@ucdavis.edu (J.P. Crutchfield), cellison@wisc.edu (C.J. Ellison), pmriechers@ucdavis.edu (P.M. Riechers).[14], and even in the formation of social structure [15], to mention several recent examples.Thus, measures of complexity are important to quantifying how organized nonlinear systems are: their randomness and their structure. Moreover, we now know that randomness and structure are intimately intertwined. One cannot be properly defined or even practically measured without the other [16, and references therein].Measuring complexity has been a challenge: Until recently, in understanding the varieties of organization to be captured; still practically, in terms of estimating metrics from experimental data. One major rea...

Informational and Causal Architecture of Continuous-time Renewal Processes

“…[1][2][3]. Some progress has been made on understanding minimal maximally predictive models of discrete-time, continuous-output processes; e.g., see Refs.…”

Maxwell Demon Dynamics: Deterministic Chaos, the Szilard Map, and the Intelligence of Thermodynamic Systems

“…The ϵ-machine for the Szilard engine is a special kind of hidden Markov model-the minimal unifilar generator-of the observed symbol sequence. Its unique properties allow for exact calculation of many essential informationtheoretic properties [29]. The ϵ-transducer is an extension that accepts control inputs and also generates outputs [30,31].…”

Predictive Rate-Distortion for Infinite-Order Markov Processes

“…This result is part of a larger body of work [36,38,57] that suggests prediction-related information properties are more accurately and more easily calculable from maximally predictive models, when available-the -machine or other prescient models [2]-than directly from trajectory distributions. These information measures are sometimes of interest to researchers, even when a model of the process is already known, because they summarize the intrinsic "uncertainty" or "predictability" of the process with a single number.…”

“…Prescient models and inferring information properties: Estimating information measures directly from sequence data encounters a curse of dimensionality or, in other words, severe undersampling. Instead, one can calculate information measures in closed-form from (derived or inferred) maximally predictive (prescient) models [38]. Rate-distortion functions are now on the list of information properties that can be accurately calculated.…”

“…(8) and the values of {λi}, δπ|W λ i |H(W A ) from the Supplementary Materials for Ref. [38]. The Even Process's total predictable information E ≈ 0.9183 bits.…”