Multi-modal convergence maps: from body schema and self-representation to mental imagery

Abstract

Understanding the world involves extracting the regularities that define the interaction of the behaving organism within this world, and computing the statistical structure characterizing these regularities. This can be based on contingencies of phenomena at various scales ranging from correlations between sensory signals (e.g., motor-proprioceptive loops) to high-level conceptual links (e.g., vocabulary grounding). Multiple cortical areas contain neurons whose receptive fields are tuned for signals co-occurring in multiple modalities. Moreover, the hierarchical organization of the cortex, described within the Convergence Divergence Zone framework, defines an ideal architecture to extract and make use of contingency at increasing levels of complexity. We present an artificial neural network model of the early cortical amodal computations, which we have demonstrated on the humanoid robot iCub. This model explains and predicts findings in neurophysiology and neuropsychology along with being an efficient tool to control the robot. In particular, through exploratory use of the body, the system learns a form of body schema in terms of specific modalities (e.g., arm proprioception, gaze proprioception, vision) and their multimodal contingencies. Once multimodal contingencies have been learned, the system is capable of generating and exploiting internal representations or mental images based on inputs in one of these multiple dimensions. The system thus provides insight on a possible neural substrate for mental imagery within the context of multimodal convergence.

Deep unsupervised network for multimodal perception, representation and classification

“…This is more powerful than representations obtained by self-organizing maps, as in [57,40], which provide a symbolic representation through the most active unit, but for which discrimination between two similar stimuli corresponding to the same unit would use the distance in the original space. Interestingly, our approach relates to the ConvergenceDivergence Zones framework [55], which states that high level representations are not copies of raw perception, but are instead the minimal record needed to reconstruct the approximation of original perceptions in the early cortices.…”

“…Another property of the network is its capacity to infer one modality given the other one, as in [40], which is one of the core properties of ConvergenceDivergence Zones [55]. Given one modality alone, the network is able to infer a classification and a parametrization which can be used to reconstruct the missing modality.…”

“…This association can then be used to influence the selforganization of monomodal maps [46]. The use of self-organizing maps to learn multimodal associations has been studied by several authors [88,68,35,71,84,40]. If these architectures are good at learning crossmodal associations, they suffer from the curse of dimensionality: projecting high dimensional data to two or three dimensions by preserving local topology (on which usually rely crossmodal associations) is difficult.…”

“…If these architectures are good at learning crossmodal associations, they suffer from the curse of dimensionality: projecting high dimensional data to two or three dimensions by preserving local topology (on which usually rely crossmodal associations) is difficult. A hierarchy of self-organizing maps is used in [40] to reduce the dimensionality of input, using the coordinates of the most active unit of each monomodal map as input of the multimodal map. In this case, two similar monomodal stimuli have to be represented by two units close enough to each other, which is problematic when the underlying physical phenomenon is high dimensional (for instance the visual appearance of the hand which is determined by 9 degrees of freedom for the iCub humanoid robot they use).…”

Just Imagine! Learning to Emulate and Infer Actions with a Stochastic Generative Architecture

“…Finally, we discuss current challenges and future application options in Section 6. Lallee and Dominey (2013) implemented a model that integrates low-level sensory data of an iCub robot, encoding multimodal contingencies in a single, 3D, and self-organizing competitive map. When driven by a single modal stimulus, this multimodal integration enables mental imagery of corresponding perceptions in other modalities.…”