Multimodal character viewpoint in quoted dialogue sequences

Abstract

We investigate the multimodal production of character viewpoint in spoken American English narratives by performing complementary qualitative and quantitative analyses of two quoted dialogues, focusing on the storyteller's use of character viewpoint gestures, character intonation, character facial expression, spatial orientation and gaze. A micro-analysis revealed that the extent of multimodal articulation depends on (i) the quoted speaker, with different multimodal articulatory patterns found for quotes by the speaker's past self vs. a third-person character, and (ii) the position of the quoted utterance within the quoted dialogue, with mid-dialogue utterances garnering less co-articulation than initial or final utterances within the quoted dialogue. We further investigated these observations using a quantitative approach, which was based on generalized additive modeling (GAM). The GAM analysis revealed different multimodal patterns for each quoted character, as indicated by the number of co-produced multimodal articulators. These patterns were found to hold regardless of the quote's position within the narrative. We discuss these findings with respect to previous work on multimodal quotation.

The challenges of viewpoint-taking when learning a sign language: Data from the ‘frog story’ in British Sign Language

“…But personal narratives, by drawing on events that the narrator has directly experienced, might potentially make it less challenging for learners of sign to take on character viewpoint. Stec et al (2017), in a micro-analysis of one personal spoken narrative involving two characters, showed that in the quoted dialogue sequences the narrator used multi-modal resources such as facial expression, manual gesture, eye gaze and posture change (including head movements) to distinguish between the two characters. It might be the case that transferring these resources from speech to sign is less challenging in the context of a personal narrative than when relating a previously unfamiliar story.…”

“…For example, facial gestures to show emotional affect -such as expressions of surprise, pain or sympathy -might accompany the telling of the unfortunate cyclist's fall. Likewise, when reporting dialogue between two or more characters, speakers have been reported to use facial expression, eye gaze, and torso position, in addition to manual gestures (Clark, 2016;Stec et al, 2017). While co-speech gesture is a rich resource for speakers to use when recounting a narrative, using articulators beyond the voice to portray viewpoints is optional and not conventionalized (i.e., speech can describe viewpoint shifting without gesture, and there is no grammar constraining how gesture should or should not be used).…”

Storytelling as a fundamental form of acting.

“…It is during these direct quotations that we expect that the characters of the story should be depicted and that proto-acting should occur. Stec et al (2015Stec et al ( , 2016Stec et al ( , 2017 analyzed the personal, semispontaneous narratives of speakers and annotated the various depictions that occurred during quoted speech. These depictions, including character intonation, are proposed to help portray the individuals being quoted by the speaker, whether that be themselves or another individual.…”

“…Previous experimental work on the vocal portrayal of individuals, real or fictional, has mainly provided qualitative information about the performance but has generally lacked quantitative prosodic details (Bavelas, Gerwing, & Healing, 2014; Bretherton, 1989; Cohen, 2011; Doukhan, Rilliard, Rosset, Adda-Decker, & D’Alessandro, 2011; Sawyer, 1996; Stec, Huiskes, & Redeker, 2015, 2016; Stec, Huiskes, Wieling, & Redeker, 2017). When quantitative prosodic details are provided, they are rarely clearly linked to character types.…”