Language as a multimodal phenomenon: implications for language learning, processing and evolution

Abstract

Our understanding of the cognitive and neural underpinnings of language has traditionally been firmly based on spoken Indo-European languages and on language studied as speech or text. However, in face-to-face communication, language is multimodal: speech signals are invariably accompanied by visual information on the face and in manual gestures, and sign languages deploy multiple channels (hands, face and body) in utterance construction. Moreover, the narrow focus on spoken Indo-European languages has entrenched the assumption that language is comprised wholly by an arbitrary system of symbols and rules. However, iconicity (i.e. resemblance between aspects of communicative form and meaning) is also present: speakers use iconic ges- tures when they speak; many non-Indo-European spoken languages exhibit a substantial amount of iconicity in word forms and, finally, iconicity is the norm, rather than the exception in sign languages. This introduction provides the motivation for taking a multimodal approach to the study of language learning, processing and evolution, and discusses the broad implications of shifting our current dominant approaches and assumptions to encompass multimodal expression in both signed and spoken languages

The problem of multimodal concurrent serial order in behavior

“…As a means of communication that has been, and still is, co-evolving with embodied agents (Christiansen and Chater, 2008), language in the wild is essentially multimodal and concurrent (Vigliocco, Perniss, and Vinson, 2014;Hilliard, O'Neal, Plumert, and Wagner, 2015). In its "default" spoken form, it appears that voice dominates over gesture and articulation over prosody -an impression that is strengthened by the possibility of capturing much of the meaning of an utterance by transcribing merely the sequence of words that comprise it (a step that Edelman (2008a, sec.…”

Language as Description, Indication, and Depiction

“…For example, researchers from the field of gesture studies have investigated how to classify and analyze different types of co-speech gestures (e.g., McNeill, 1992; Goldin-Meadow, 2003; Streeck, 2009), including the identification of different types of gestures with respect to their function and degrees of conventionalization and grammaticalization (e.g., Kendon, 2004; Wilcox, 2007; Calbris, 2011; for an overview see Müller et al, 2013, 2014, especially Bressem, 2013). Signed language linguists have investigated the coordination of different types of signs and strategies for making meaning used by deaf signers (e.g., Sutton-Spence and Woll, 1999; Liddell, 2003; Johnston, 2012; Vigliocco et al, 2014), including recent efforts to directly compare the communication of deaf signers with hearing co-speech gesture (see Perniss et al, 2015, inter alia ).…”

Multimodal analysis of quotation in oral narratives

“…For example, when adopting character perspective in sign languages, there is evidence that the strategies used to enact two or more characters with contrastive roles (Padden 1986) differ from the strategies used to enact characters elsewhere (Janzen 2012). As speakers and signers share certain capacities for iconic representation Vigliocco et al 2014), it might be the case that speakers use different multimodal articulators when a different number of characters are involved. Second, the number of quoted utterances could affect the multimodal strategies used.…”