Do we really need a Multimodal Construction Grammar?

Abstract

A multimodal construction is said to be a conventional pairing of a complex form, comprising at least a verbal and a kinetic element, with a specific meaning or a specific function. Do we need a new constructional approach to account for such multimodal constructions? What are the challenges to account for multimodality? The aim of this contribution is to provide a precise notion ‘multimodal construction’ and, on this basis, to indicate possible pathways for future investigations. The paper opts for cautiously extending the scope of existing constructional approaches in order to include non-linguistic meaningful behavior. In particular, it is argued that even though Construction Grammar invites for treating multimodal on a par with linguistic constructions, there is a huge lack of substantial empirical support to arrive at a more detailed and data-based understanding of the nature of multimodal constructions.

Quantifying the speech-gesture relation with massive multimodal datasets: Informativity in time expressions

“…There is simply so much we do not know about how multimodal information comes together to signal for something in particular. Could multimodal features also be part of a linguistic pattern or grammatical construction [1][2][3][4][5][6][7][8][9]? Can non-verbal modalities be pervasively and deeply structured in a way that compares with language?…”

Construction grammar for monkeys?

“…In recent years, construction grammarians have started to take phenomena into account that have previously been treated as epiphenomenal at best, e.g. multimodal constructions such as conventionalised co-speech gestures (Zima, 2014, Cienki, 2017 or memes (Dancygier & Vandelanotte, 2017). This also shows that language is not a nicely delineated, clear-cut entity, which fits in with the assumption of a continuum between non-linguistic and linguistic communication.…”

“…As research on multimodal construction grammar has shown, many constructions integrate not only vocal-auditory, but also gestural and other modalities, calling for an integrated system of speech, gesture, and other modalities (e.g. Steen & Turner, 2013;Ziem, 2017;Hall et al, 2019). Given that primate communication is also fundamentally multimodal in nature to a large degree (e.g.…”

When Gesture “Takes Over”: Speech-Embedded Nonverbal Depictions in Multimodal Interaction

“…Finally, recent years have seen attempts at incorporating gesture into the theoretical framework of linguistic analysis, coming from various theoretical orientations and with different approaches [e.g., “integrated message model” ( Bavelas and Chovil, 2000 ); “composite signal” ( Clark, 1996 ); “composite utterance” ( Enfield, 2009 ); “multimodal grammar” ( Fricke, 2012 ); multimodal negation ( Harrison, 2018 ); incorporation of gesture into Cognitive Grammar ( Kok and Cienki, 2016 ); “mixed syntax” ( Slama-Cazacu, 1976 )]. Construction Grammar, in particular, sees a recent debate on Multimodal Construction Grammar (e.g., Steen and Turner, 2013 ; Schoonjans et al, 2015 ; Cienki, 2017 ; Hoffmann, 2017 ; Schoonjans, 2017 ; Ziem, 2017 ; Zima and Bergs, 2017 ). Arguing for nonverbal signals being as integral to language as canonical speech, these studies touch upon cases of gestures without simultaneous speech, acknowledging their crucial role in language use, but the primary focus remains on gesture-speech co-occurrence.…”