Real-time detection and interpretation of 3D deictic gestures for interactionwith an intelligent environment

Abstract

We present a system that enables pointing-based unconstrained interaction with a smart conference room using an arbitrary multi-camera setup. For each individual camera stream, areas exhibiting strong motion are identified. In these areas, face and hand hypotheses are detected. The detections of multiple cameras are then combined to 3D hypotheses from which deictic gestures are identified and a pointing direction is derived. This is then used to identify objects in the scene. Since we use a combination of simple yet effective techniques, the system runs in real-time and is very responsive. We present evaluation results on realistic data that show the capabilities of the presented approach.

Saliency-based identification and recognition of pointed-at objects

“…An often utilized approach is to calculate the direction as the line-of-sight between the eyes and the pointing hand or finger (e.g. [37], [40]). In [39], 3 different possibilities were evaluated, and the line-of-sight model was reported to be the best.…”

“…Saliency-based Object Detection and Selection 1) In order to detect pointing gestures, we use a modified version of the system presented in [40]. Most importantly, we replaced the face detector with a head-shoulder detector based on histograms of oriented gradients, which -in practice -is more robust and less view-dependent.…”

“…Since we do not specifically evaluate the pointing detector (cf. [40]), we discard cases with erroneous pointing gesture detections. Thus, in total, our evaluation set contains 220 object references.…”

Gesture-Based Interaction in Domotic Environments: State of the Art and HCI Framework Inspired by the Diversity

“…Some solutions acknowledge this principle [13,29], but do not take into account whether the gestures are ambiguous regarding the target population, since ambiguity is intrinsically related to the cultural aspects of the population. As an example of the problem of not considering the interdependency of gestural ambiguity with cultural aspects, the application in [39] uses only deictic gestures (pointing gestures). Gestures for some commands for this application resemble a firearm, which is probably not desired in a home or in war-or conflict-ridden regions of the world.…”

Focusing computational visual attention in multi-modal human-robot interaction

“…Accordingly, visually recognizing pointing gestures and inferring a referent or target direction has been addressed by several authors; e.g., for interaction with smart environments (e.g. [39]), wearable visual interfaces (e.g. [22]), and robots (e.g.…”