The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review

Abstract

Human language is inherently embodied and grounded in sensorimotor representations of the self and the world around it. This suggests that the body schema and ideomotor action-effect associations play an important role in language understanding, language generation, and verbal/physical interaction with others. There are computational models that focus purely on non-verbal interaction between humans and robots, and there are computational models for dialog systems that focus only on verbal interaction. However, there is a lack of research that integrates these approaches. We hypothesize that the development of computational models of the self is very appropriate for considering joint verbal and physical interaction. Therefore, they provide the substantial potential to foster the psychological and cognitive understanding of language grounding, and they have significant potential to improve human-robot interaction methods and applications. This review is a first step toward developing models of the self that integrate verbal and non-verbal communication. To this end, we first analyze the relevant findings and mechanisms for language grounding in the psychological and cognitive literature on ideomotor theory. Second, we identify the existing computational methods that implement physical decision-making and verbal interaction. As a result, we outline how the current computational methods can be used to create advanced computational interaction models that integrate language grounding with body schemas and self-representations.

Disembodied Language in Early-Onset Schizophrenia

“…Empirical studies using EASE revealed a significant coincidence of self-disorder with typical formal thought aberrancies and imagination anomalies ( 34 ), which suggests the crucial role of the self in language processing. An embodied approach to language defines concepts as having the same structure as perceptions, such as the color, shape, movement, and emotional value ( 35 ). In the study, we have used spatial properties.…”

Spatial relation learning in complementary scenarios with deep neural networks

“…While RL is established for learning physical actions and suitable for general use (Silver et al, 2021 ), its use for language learning is yet emergent (Röder et al, 2021 ; Uc-Cetina et al, 2021 ). Regarding language as a sequence production problem, our language decoder could benefit from the availability of high-quality forward models, such as the Transformer language model.…”