Motion verbs in multimodal communication

Abstract

The article explores correlations between motion verbs and head and hands gestures using the RUPEX corpus. The verbs are divided into four groups based on their meanings. Мonological and dialogical parts of the recordings are compared along with the speaker’s role and viewpoint in gestures. The pilot analysis of motion verbs in the multimodal corpus showed that the relationships between verb type, non-verbal behavior and speaker’s role depend on a complex set of factors and manifests itself in different ways in different channels. In the verbal channel no direct relationship between the semantic type of the verb and the speaker’s role was detected; however, the narrators and commentators who have seen the film used more affectional vocabulary than the reteller while the latter tended to use more vector-prefixed verbs. In manual channel рrefixes or their absence do not influence the use of hand gestures. Transitive verbs meaning manipulations of different items are more probable to be illustrated by depictive gestures. Predictably, motion verbs in the strict sense are more prone to be supported by observer viewpoint (O-VPT) gestures, while verbs of manipulation are usually used with C-VPT gestures. In cephalic channel motion verbs in the strict sense (relocation of a character) are usually illustrated by O-VPT depictive gestures, and manipulation verbs are more probably supported by pantomime C-VPT gestures similar to manual channel. In some head gestures the viewpoint is combined. If the verb is repeated by the same or another speaker the gestures differ in both manual and cephalic channels. Cephalic gesture clusters on motion verbs have mostly a depictive function, which may be considered a gestural illustration.

Vague reference in expository discourse: multimodal regularities of speech and gesture

“…Figures 1 and 2 manifest that annotations allowed to synchronize the use of discourse markers and gestures. As seen in Figures1 and 2, the discourse segment for speech analysis was the minimal discourse unit identified following both prosodic and syntactic criteria; most commonly it corresponds to a clause[28] manifested here in не знаю точно их различие (Figure1) and in тебе кто-то их назначает (Figure2). The discourse unit under consideration in Figure1displays three gesture uses which are pragmatic gestures (coded as 2305 and 2303) and one adaptor (coded as 2401); still the discourse marker of vague reference (coded as 1203) is synchronized only with the first pragmatic gesture (coded as 2305).…”