Untethered gesture acquisition and recognition for virtual world manipulation

Abstract

Humans use a combination of gesture and speech to interact with objects and usually do so more naturally without holding a device or pointer. We present a system that incorporates user body-pose estimation, gesture recognition and speech recognition for interaction in virtual reality environments. We describe a vision-based method for tracking the pose of a user in real time and introduce a technique that provides parameterized gesture recognition. More precisely, we train a support vector classifier to model the boundary of the space of possible gestures, and train Hidden Markov Models on specific gestures. Given a sequence, we can find the start and end of various gestures using a support vector classifier, and find gesture likelihoods and parameters with a HMM. A multimodal recognition process is performed using rank-order fusion to merge speech and vision hypotheses. Finally we describe the use of our multimodal framework in a virtual world application that allows users to interact using gestures and speech.

Human Robot Interaction Through Semantic Integration of Multiple Modalities, Dialog Management, and Contexts

“…Table 1.1 illustrates how using multiple modalities, dialog management, context, and semantics improves recognition. Except for the research of Demirdjian et al [25], those using multiple HCI techniques have lower error rates than those using only a single HCI technique. Even the research of Demirdjian et al produced lower error rates when multiple techniques were used.…”

“…Error Rate Demirdjian et al [25] Vision & speech 0% Demirdjian et al [25] Speech 5% Demirdjian et al [25] Vision 8% Morency et al [61] Gesture & dialog context 8% Morency and Darrell [60] Gestures & dialog state 9% Quattoni et al [67] Vision & semantics 9% Wang and Demirdjian [86] Speech & gestures 12% Webb et al [87] Speech & dialog state 17% Metze et al [59] Speech, context, & gesture 17% Morency et al [61] Gesture 22% Saenko et al [72] Vision 34% Eisenstein and Davis [31] Linguistic context 34% Bugmann [13] Speech 40% However, these techniques have not been applied to the same extent in Human…”

“…A variety of contexts, such as domain context and conversation context, were also used to enhance multi-modal interpretation. In a telephone banking call center application, Demirdjian et al describe a virtual studio application that allows a user to edit a virtual world and navigate around it [25]. The application uses speech, one-handed, and two-handed gestures to identify ambiguous references in speech.…”

Gestural Interaction for Virtual Reality Environments through Data Gloves

“…From these works, it has been identified the following specified shortcomings [3,4,5,6,7,8,9,10,11,12,13,14,15]:…”

Human pose estimation from a single view point, real-time range sensor

“…In [17], control theory is used to maintain a correspondence between model based feature points and depth points. In [18], Iterative Closest Point (ICP) is used to track a pose initialized using a hashing method. In [19], information in depth images and silhouettes is used in a learning framework to infer poses from a database.…”