Multimodal human discourse

Abstract

Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the gesture research done to date, and present our work on the cross-modal cues for discourse segmentation in free-form gesticulation accompanying speech in natural conversation as a new paradigm for such multimodal interaction. The basis for this integration is the psycholinguistic concept of the coequal generation of gesture and speech from the same semantic intent. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. We perform F. Quek et al.two independent sets of analyses on the video and audio data: video and audio analysis to extract segmentation cues, and expert transcription of the speech and gesture data by microanalyzing the videotape using a frame-accurate videoplayer to correlate the speech with the gestural entities. We compare the results of both analyses to identify the cues accessible in the gestural and audio data that correlate well with the expert psycholinguistic analysis. We show that "handedness" and the kind of symmetry in two-handed gestures provide effective supersegmental discourse cues.

Eigengestures for Natural Human Computer Interface

“…Additionally, each gesture was made once at a rapid pace and once at a slow pace. Gestures number 2,3,7,8,10,12,13,14,15,17,18,19 are periodical and in their case the Table 1. The gesture list prepared with the proposed methodology.…”

“…The gesture list prepared with the proposed methodology. Notes: a-We use the terms 'symbolic', 'deictic', and 'iconic' based on McNeill & Levy [8] classification, supplemented with a category of 'manipulative' gestures (following [10]), b-Significant motion components: T-hand translation, R-hand rotation, F-individual finger movement, c-This gesture is usually accompanied with a specific object (deictic) reference.…”

Layered Hypernetwork Models for Cross-Modal Associative Text and Image Keyword Generation in Multimodal Information Retrieval

“…In HCI, various modalities are studied including speeches and gestures. Quek et al studied multimodal human discourse in aspect of gesture and speech [12]. Christoudias et al proposed cotraining method of multimodal data to construct multimodal interface [13].…”

“…According to (12), hyperedges with unique information get higher weights by definition. Also, hyperedges with low weight values are eliminated and the erased amounts of hyperedges are regenerated from training set.…”

GPU-accelerated computation for robust motion tracking using the CUDA framework

“…Vector Coherence Mapping (VCM) is a video motion tracking algorithm first introduced by Quek et al [1,7,8], and has been applied to a wide variety of gesture analysis [9,10,12]. VCM is an inherently parallel algorithm that tracks sets of interest points as a spatially and temporally coherent vector fields from raw video.…”