The development of large-scale corpora has led to a quantum leap in our understanding of speech in recent years. By contrast, the analysis of massive datasets has so far had a limited impact on the study of gesture and other visual communicative behaviors. We utilized the UCLA-Red Hen Lab multi-billion-word repository of video recordings, all of them showing communicative behavior that was not elicited in a lab, to quantify speech-gesture co-occurrence frequency for a subset of linguistic expressions in American English. First, we objectively establish a systematic relationship in the high degree of co-occurrence between gesture and speech in our subset of expressions, which consists of temporal phrases. Second, we show that there is a systematic alignment between the informativity of co-speech gestures and that of the verbal expressions with which they co-occur. By exposing deep, systematic relations between the modalities of gesture and speech, our results pave the way for the data-driven integration of multimodal behavior into our understanding of human communication.

No abstract

This study investigates how typological and metaphorical construal differences may affect the use and frequency of temporal expressions in English and Spanish. More precisely, we explore whether there are any differences between English, a satelliteframed language, and Spanish, a verb-framed language, in the use of certain temporal linguistic expressions that include a spatial, deictic component (Deictic Time), a purely temporal relation between two events (Sequential Time) or the expression of the duration of an event (Duration). To achieve this, we perform two different types of studies. First, we conduct an informational gain or loss analysis of 1,650 of Englishto-Spanish translations extracted from parallel corpora. Secondly, we compare the frequency of 33 English and 27 Spanish temporal expressions in two similar written online corpora (EnTenTen and EsTenTen, respectively) and a television news spoken corpus (NewsScape). Our results suggest that English uses "deictic expressions with directional language" (explicitly stating the spatial location of the temporal event, e.g., back in those days/in the future ahead) much more frequently than Spanish, to the extent that such directional information is often excluded in English-to-Spanish translations. Also, sequential expressions (such as before that/later than) and duration expressions (during the whole day) are much more frequent in Spanish. These usage differences, explained by the variability in motion typology and metaphoric construal, open up the interesting question of how these differences in linguistic usage could affect the conceptualization of time of English and Spanish speakers.

No abstract

There is a distinction between languages that use the duration is length metaphor, like English (e.g., long time), and languages like Spanish that conceptualise time using the duration is quantity metaphor (e.g., much time). The present study examines the use of both metaphors, exploring their multimodal behaviour in Spanish speakers. We analyse co-speech gesture patterns in the TV news setting, using data from the NewsScape Library, that co-occur with expressions that trigger the duration is quantity construal (e.g., durante todo ‘during the whole’) and the duration is length construal in the from X to Y construction (e.g., desde el principio hasta el final ‘from beginning to end’). Results show that both metaphors tend to co-occur with a semantic gesture, with a preference for the lateral axis, as reported in previous studies. However, our data also indicate that the direction of the gesture changes depending on the construal. The duration is quantity metaphor tends to be performed with gestures with an outwards direction, in contrast with the duration is length construal, which employ a left-to-right directionality. These differences in gesture realisation point to the existence of different construals for the concept of temporal duration.