A Compositional Neural Architecture for Language

Abstract

Hierarchical structure and compositionality imbue human language with unparalleled expressive power and set it apart from other perception–action systems. However, neither formal nor neurobiological models account for how these defining computational properties might arise in a physiological system. I attempt to reconcile hierarchy and compositionality with principles from cell assembly computation in neuroscience; the result is an emerging theory of how the brain could convert distributed perceptual representations into hierarchical structures across multiple timescales while representing interpretable incremental stages of (de)compositional meaning. The model's architecture—a multidimensional coordinate system based on neurophysiological models of sensory processing—proposes that a manifold of neural trajectories encodes sensory, motor, and abstract linguistic states. Gain modulation, including inhibition, tunes the path in the manifold in accordance with behavior and is how latent structure is inferred. As a consequence, predictive information about upcoming sensory input during production and comprehension is available without a separate operation. The proposed processing mechanism is synthesized from current models of neural entrainment to speech, concepts from systems neuroscience and category theory, and a symbolic-connectionist computational model that uses time and rhythm to structure information. I build on evidence from cognitive neuroscience and computational modeling that suggests a formal and mechanistic alignment between structure building and neural oscillations, and moves toward unifying basic insights from linguistics and psycholinguistics with the currency of neural computation.

Commentary: A Compositional Neural Architecture for Language

“…Martin ( 2020 ) defends a multidimensional coordinate system for language based on neurophysiological models of sensory processing. With ambitious scope, Martin aims to connect theories of processing and neural implementation, an extraordinarily challenging undertaking.…”

“…Martin ( 2020 ) reports how gain modulation underlies “coordinate transform between sensory modalities and between sensory and motor systems.” She cites classic work on cellular dynamics (Salinas and Abbott, 2001 ), but does not cite other work (e.g., Dayan and Abbott, 2001 ) where the limitations of gain modulation are stressed, e.g., gaze-dependent gain modulation of retinotopic visual receptive fields is well-reported, but the relevance of gain modulation to higher cognition is not established [(Dayan and Abbott, 2001 ), p. 17].…”

“…Martin ( 2020 ) makes an additional claim that “gain modulation also offers a built-in system for predictive coding,” citing Friston ( 2005 ); more recent, updated accounts (e.g., Friston, 2018 ) rely on an interplay of a range of other processes. Moreover, Friston's ventures into language (i.e., Friston et al, 2017 ) rely on other neural mechanisms (belief propagation in neuronal process theories, phase precession, theta-gamma coupling) employing distinct softmax functions, logarithmic transforms, and linear algebra (implemented via firing rate functions, non-linear postsynaptic responses, and neuronal connectivity, respectively).…”

“…Further, regulation of neural gain can provide “an integration mechanism whereby information from multiple sources can be non-linearly combined via multiplicative modulation of the cell's response to inputs” [Ferguson and Cardin, 2020 , p. 81]. Martin ( 2020 ) defends an additive, rather than a tensor-based multiplicative, formulation of linguistic compositionality. Ferguson and Cardin ( 2020 ) review how visual properties that must be decoded separately in higher-order areas of non-human primate visual cortex are combined multiplicatively (e.g., object identity), whereas parts that must be integrated (e.g., object sub-parts) are combined additively.…”

“…Martin ( 2020 ) defends a model of linguistic computation relying on gain modulation. I will argue that any gain modulation model of language faces a number of internal conceptual and empirical difficulties, and that our current understanding of gain modulation does not support its use as a core component of linguistic computation.…”