Quantifying Information Modification in Developing Neural Networks via Partial Information Decomposition

Wibral, Michael; Finn, Conor; Wollstadt, Patricia; Lizier, Joseph T.

doi:10.3390/e19090494

Cited by 60 publications

(87 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, this would certainly be a worthwhile exercise before attempting to generalise the information decomposition to continuous random variables. Regarding future empirical work, there are many rich data sets which could be decomposed using this decomposition including financial time-series and neural recordings, e.g., [ 28 , 33 , 34 ].…”

Section: Discussionmentioning

confidence: 99%

“…In addition to these proposed replacements for

, there is also a substantial body of literature discussing either PID, similar attempts to decompose multivariate information, or the problem of information decomposition in general [ 3 , 4 , 5 , 7 , 10 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 ]. Furthermore, the current proposals have been applied to various problems in neuroscience [ 29 , 30 , 31 , 32 , 33 , 34 ]. Nevertheless (to date), there is no generally accepted measure of redundant information that is entirely compatible with PID framework, nor has any other well-accepted multivariate information decomposition emerged.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Pointwise Partial Information Decomposition Using the Specificity and Ambiguity Lattices

Finn

Lizier

2018

Entropy

Self Cite

183

View full text Add to dashboard Cite

What are the distinct ways in which a set of predictor variables can provide information about a target variable? When does a variable provide unique information, when do variables share redundant information, and when do variables combine synergistically to provide complementary information? The redundancy lattice from the partial information decomposition of Williams and Beer provided a promising glimpse at the answer to these questions. However, this structure was constructed using a much criticised measure of redundant information, and despite sustained research, no completely satisfactory replacement measure has been proposed. In this paper, we take a different approach, applying the axiomatic derivation of the redundancy lattice to a single realisation from a set of discrete variables. To overcome the difficulty associated with signed pointwise mutual information, we apply this decomposition separately to the unsigned entropic components of pointwise mutual information which we refer to as the specificity and ambiguity. This yields a separate redundancy lattice for each component. Then based upon an operational interpretation of redundancy, we define measures of redundant specificity and ambiguity enabling us to evaluate the partial information atoms in each lattice. These atoms can be recombined to yield the sought-after multivariate information decomposition. We apply this framework to canonical examples from the literature and discuss the results and the various properties of the decomposition. In particular, the pointwise decomposition using specificity and ambiguity satisfies a chain rule over target variables, which provides new insights into the so-called two-bit-copy example.

show abstract

Section: Discussionmentioning

confidence: 99%

“…In addition to these proposed replacements for

Section: Introductionmentioning

confidence: 99%

Pointwise Partial Information Decomposition Using the Specificity and Ambiguity Lattices

Finn

Lizier

2018

Entropy

Self Cite

183

View full text Add to dashboard Cite

show abstract

“…Wibral et al [ 43 ] apply PID to decompose information storage, transfer and in particular information modification in developing neural cultures, following the perspective of Lizier et al [ 7 ]. Utilising the Bertschinger et al [ 12 ] PID measure via the publicly available IDT

toolkit [ 44 ], the authors identify the aforementioned components of information processing from pairs of input (multi-unit) spike train recordings to each output recording.…”

Section: Contents Of the Special Issuementioning

confidence: 99%

Information Decomposition of Target Effects from Multi-Source Interactions: Perspectives on Previous, Current and Future Work

Lizier

Bertschinger

Jost

et al. 2018

Entropy

Self Cite

133

106

View full text Add to dashboard Cite

Abstract:The formulation of the Partial Information Decomposition (PID) framework by Williams and Beer in 2010 attracted a significant amount of attention to the problem of defining redundant (or shared), unique and synergistic (or complementary) components of mutual information that a set of source variables provides about a target. This attention resulted in a number of measures proposed to capture these concepts, theoretical investigations into such measures, and applications to empirical data (in particular to datasets from neuroscience). In this Special Issue on "Information Decomposition of Target Effects from Multi-Source Interactions" at Entropy, we have gathered current work on such information decomposition approaches from many of the leading research groups in the field. We begin our editorial by providing the reader with a review of previous information decomposition research, including an overview of the variety of measures proposed, how they have been interpreted and applied to empirical investigations. We then introduce the articles included in the special issue one by one, providing a similar categorisation of these articles into: i. proposals of new measures; ii. theoretical investigations into properties and interpretations of such approaches, and iii. applications of these measures in empirical studies. We finish by providing an outlook on the future of the field.Keywords: mutual information; information decomposition; unique information; redundant information; complementary information; redundancy; synergy PACS: 89.70.Cf; 89.75.Fb; 05.65.+b; 87.19.lo Background to Information DecompositionShannon information theory [1][2][3] has provided rigorous ways to capture our intuitive notions regarding uncertainty and information, and it has made an enormous impact in doing so. One of the fundamental measures here is mutual information I(S; T), which captures the average information contained in samples s of a set of source variables S about samples t of another variable T, and vice versa. If we have two source variables S 1 , S 2 and a target T, for example, we can measure:1.the information held by one source about the target I

show abstract

“…The partial information decomposition has been applied to data in neuroscience; see, e.g. [46,47,48,49]. For a recent overview, see [50].…”

Section: Application Of Partial Information Decompositionmentioning

confidence: 99%

Bayesian modeling of BAC firing as a mechanism for apical amplification in neocortical pyramidal neurons

Kay¹,

Phillips

Aru

et al. 2019

Preprint

View full text Add to dashboard Cite

Pyramidal cells in layer 5 of the neocortex have two distinct integration sites. These cells integrate inputs to basal dendrites in the soma while integrating inputs to the tuft in a site at the top of the apical trunk. The two sites communicate by action potentials that backpropagate to the apical site and by backpropagation-activated calcium spikes (BAC firing) that travel from the apical to the somatic site. Six key messages arise from the probabilistic information-theoretic analyses of BAC firing presented here. First, we suggest that pyramidal neurons with BAC firing could convert the odds in favour of the presence of a feature given the basal data into the odds in favour of the presence of a feature given the basal data and the apical input, by a simple Bayesian calculation. Second, the strength of the cell's response to basal input can be amplified when relevant to the current context, as specified by the apical input, without corrupting the message that it sends. Third, these analyses show rigorously how this apical amplification depends upon communication between the sites. Fourth, we use data on action potentials from a very detailed multi-compartmental biophysical model to study our general model in a more realistic setting, and demonstrate that it describes the data well. Fifth, this form of BAC firing meets criteria for distinguishing modulatory from driving interactions that have been specified using recent definitions of multivariate mutual information. Sixth, our general decomposition can be extended to cases where, instead of being purely driving or purely amplifying, apical and basal inputs can be partly driving and partly amplifying to various extents. These conclusions imply that an advance beyond the assumption of a single site of integration within pyramidal cells is needed, and suggest that the evolutionary success of neocortex may depend upon the cellular mechanisms of context-sensitive selective amplification hypothesized here. Author summaryThe cerebral cortex has a key role in conscious perception, thought, and action, and is predominantly composed of a particular kind of neuron: the pyramidal cells. The distinct shape of the pyramidal neuron with a long dendritic shaft separating two regions of profuse dendrites allows them to integrate inputs to the two regions separately and combine the results non-linearly to produce output. Here we show how inputs to this more distant site strengthen the cell's output 1 when it is relevant to the current task and environment. By showing that such neurons have capabilities that transcend those of neurons with the single site of integration assumed by many neuroscientists, this 'splitting of the neuronal atom' offers a radically new viewpoint from which to understand the evolution of the cortex and some of its many pathologies. This also suggests that approaches to artificial intelligence using neural networks might come closer to something analogous to real intelligence, if, instead of basing them on processing elements with a single site of integra...

show abstract

Quantifying Information Modification in Developing Neural Networks via Partial Information Decomposition

Cited by 60 publications

References 49 publications

Pointwise Partial Information Decomposition Using the Specificity and Ambiguity Lattices

Pointwise Partial Information Decomposition Using the Specificity and Ambiguity Lattices

Information Decomposition of Target Effects from Multi-Source Interactions: Perspectives on Previous, Current and Future Work

Bayesian modeling of BAC firing as a mechanism for apical amplification in neocortical pyramidal neurons

Contact Info

Product

Resources

About