Efficient representation as a design principle for neural coding and computation

Bialek, William; Steveninck, R.Rd.R. van; Tishby, Naftali

doi:10.1109/isit.2006.261867

Cited by 69 publications

(66 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although there has been much interest in the brain's ability to predict particular things, our approach emphasizes that prediction is a general problem, which can be stated in a unified mathematical structure across many contexts, from the extrapolation of trajectories to the learning of rules (20). Our results on the efficient representation of predictive information in the retina thus may hint at a much more general principle.…”

Section: Discussionmentioning

confidence: 94%

Predictive information in a sensory population

Palmer

Marre

Berry

et al. 2015

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

228

276

View full text Add to dashboard Cite

Guiding behavior requires the brain to make predictions about the future values of sensory inputs. Here, we show that efficient predictive computation starts at the earliest stages of the visual system. We compute how much information groups of retinal ganglion cells carry about the future state of their visual inputs and show that nearly every cell in the retina participates in a group of cells for which this predictive information is close to the physical limit set by the statistical structure of the inputs themselves. Groups of cells in the retina carry information about the future state of their own activity, and we show that this information can be compressed further and encoded by downstream predictor neurons that exhibit feature selectivity that would support predictive computations. Efficient representation of predictive information is a candidate principle that can be applied at each stage of neural computation.neural coding | retina | information theory A lmost all neural computations involve making predictions. Whether we are trying to catch prey, avoid predators, or simply move through a complex environment, the data we collect through our senses can guide our actions only to the extent that these data provide information about the future state of the world. Although it is natural to focus on the prediction of rewards (1), prediction is a much broader problem, ranging from the extrapolation of the trajectories of moving objects to the learning of abstract rules that describe the unfolding pattern of events around us (2-4). An essential aspect of the problem in all these forms is that not all features of the past carry predictive power. Because there are costs associated with representing and transmitting information, it is natural to suggest that sensory systems have optimized coding strategies to keep only a limited number of bits of information about the past, ensuring that these bits are maximally informative about the future. This principle can be applied at successive stages of signal processing, as the brain attempts to predict future patterns of neural activity. We explore these ideas in the context of the vertebrate retina, provide evidence for nearoptimal coding, and find that this performance cannot be explained by classical models of ganglion cell firing. Coding for the Position of a Single Visual ObjectThe structure of the prediction problem depends on the structure of the world around us. In a world of completely random stimuli, for example, prediction is impossible. Consider a simple visual world such that, in the small patch of space represented by the neurons from which we record, there is just one object (a dark horizontal bar against a light background) moving along a trajectory x t . We want to construct trajectories that are predictable, but not completely; the moving object has some inertia, so that the velocities υ t are correlated across time, but is also "kicked" by unseen random forces. A mathematically tractable example (Eqs. 4 and 5 in Materials and Methods) is shown in Fig...

show abstract

Section: Discussionmentioning

confidence: 94%

Predictive information in a sensory population

Palmer

Marre

Berry

et al. 2015

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

228

276

View full text Add to dashboard Cite

show abstract

“…If one adopts the view that organisms tend to implement an information parsimony principle (Laughlin 2001;Polani 2009), then this implies that biological systems will exhibit a tendency to achieve a given level of performance at the lowest informational cost possible (or perform as well as possible under a given informational bandwidth). In our formalism, this would correspond to operating close to the optimal reward/information (strictly spoken, decision complexity) trade-off curve, always assuming that a suitable reward function can be formulated (Taylor et al 2007;Bialek et al 2007).…”

Section: Soft Vs Sharp Policiesmentioning

confidence: 99%

Information Theory of Decisions and Actions

Tishby

Polani²

2010

Perception-Action Cycle

Self Cite

220

262

View full text Add to dashboard Cite

The perception-action cycle is often defined as "the circular flow of information between an organism and its environment in the course of a sensory guided sequence of actions towards a goal" (Fuster 2001(Fuster , 2006. The question we address in this paper is in what sense this "flow of information" can be described by Shannon's measures of information introduced in his mathematical theory of communication. We provide an affirmative answer to this question using an intriguing analogy between Shannon's classical model of communication and the Perception-Action-Cycle. In particular, decision and action sequences turn out to be directly analogous to codes in communication, and their complexity -the minimal number of (binary) decisions required for reaching a goal -directly bounded by information measures, as in communication. This analogy allows us to extend the standard Reinforcement Learning framework. The latter considers the future expected reward in the course of a behaviour sequence towards a goal (value-to-go). Here, we additionally incorporate a measure of information associated with this sequence: the cumulated information processing cost or bandwidth required to specify the future decision and action sequence (information-to-go).Using a graphical model, we derive a recursive Bellman optimality equation for information measures, in analogy to Reinforcement Learning; from this, we obtain new algorithms for calculating the optimal trade-off between the value-to-go and the required information-to-go, unifying the ideas behind the Bellman and the Blahut-Arimoto iterations. This trade-off between value-to-go and information-togo provides a complete analogy with the compression-distortion trade-off in source coding. The present new formulation connects seemingly unrelated optimization problems. The algorithm is demonstrated on grid world examples.

show abstract

“…→ X] has been studied by several authors and given different names, such as (in chronological order) convergence rate of the conditional entropy [27], excess entropy [28], stored information [29], effective measure complexity [30], past-future mutual information [31], and predictive information [32], amongst others. For a review see Ref.…”

Section: Causal Statesmentioning

confidence: 99%

Optimal causal inference: Estimating stored information and approximating causal architecture

Still

Crutchfield

Ellison

2010

Chaos: An Interdisciplinary Journal of Nonlinear Science

102

View full text Add to dashboard Cite

We introduce an approach to inferring the causal architecture of stochastic dynamical systems that extends rate distortion theory to use causal shielding-a natural principle of learning. We study two distinct cases of causal inference: optimal causal filtering and optimal causal estimation.Filtering corresponds to the ideal case in which the probability distribution of measurement sequences is known, giving a principled method to approximate a system's causal structure at a desired level of representation. We show that, in the limit in which a model complexity constraint is relaxed, filtering finds the exact causal architecture of a stochastic dynamical system, known as the causal-state partition. From this, one can estimate the amount of historical information the process stores. More generally, causal filtering finds a graded model-complexity hierarchy of approximations to the causal architecture. Abrupt changes in the hierarchy, as a function of approximation, capture distinct scales of structural organization.For nonideal cases with finite data, we show how the correct number of underlying causal states can be found by optimal causal estimation. A previously derived model complexity control term allows us to correct for the effect of statistical fluctuations in probability estimates and thereby avoid over-fitting. Natural systems compute intrinsically and produce information. This organization, often only indirectly accessible to an observer, is reflected to varying degrees in measured time series. Nonetheless, this information can be used to build models of varying complexity that capture the causal architecture of the underlying system and allow one to estimate its information processing capabilities. We investigate two cases. The first is when a model builder wishes to find a more compact representation than the true one. This occurs, for example, when one is willing to incur the cost of a small increase in error for a large reduction in model size. The second case concerns the empirical setting in which only a finite amount of data is available. There one wishes to avoid over-fitting a model to a particular data set.

show abstract

Efficient representation as a design principle for neural coding and computation

Cited by 69 publications

References 19 publications

Predictive information in a sensory population

Predictive information in a sensory population

Information Theory of Decisions and Actions

Optimal causal inference: Estimating stored information and approximating causal architecture

Contact Info

Product

Resources

About