The catchment feature model for multimodal language analysis

Quek,

doi:10.1109/iccv.2003.1238394

Cited by 5 publications

(1 citation statement)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At the same time, these spatio-temporal images are embodied in human gesture accompanying speech. As such, human gesture and speech coheres, not at the lexical or syntactic level (there is no such thing as a consistent gesticulatory correlate to a particular word), but at the level of the units of meaning and discourse as they are constructed (McNeill, 2000;Quek et al, 2002;Quek, 2003). This gives rise to the psycholinguistic concept known as the catchment (McNeill, 2000a(McNeill, , 2000bMcNeill et al, 2001).…”

Section: Introductionmentioning

confidence: 99%

Hand Motion Gesture Frequency Properties and Multimodal Discourse Analysis

Xiong

Quek

2006

Int J Comput Vision

View full text Add to dashboard Cite

Abstract. Gesture and speech are co-expressive and complementary channels of a single human language system. While speech carries the major load of symbolic presentation, gesture provides the imagistic content. We investigate the role of oscillatory/cyclical hand motions in 'carrying' this image content. We present our work on the extraction of hand motion oscillation frequencies of gestures that accompany speech. The key challenges are that such motions are characterized by non-stationary oscillations, and multiple frequencies may be simultaneously extant. Also, the duration of the oscillations may be extended over very few cycles. We apply the windowed Fourier transform and wavelet transform to detect and extract gesticulatory oscillations. We tested these against synthetic signals (stationary and non-stationary) and real data sequences of gesticulatory hand movements in natural discourse. Our results show that both filters functioned well for the synthetic signals. For the real data, the wavelet bandpass filter bank is better for detecting and extracting hand gesture oscillations. We relate the hand motion oscillatory gestures detected by wavelet analysis to speech in natural conversation and apply to multimodal language analysis. We demonstrate the ability of our algorithm to extract gesticulatory oscillations and show how oscillatory gestures reveal portions of the multimodal discourse structure.

show abstract