Multi-modal and multi-camera attention in smart environments

Abstract

This paper considers the problem of multi-modal saliency and attention. Saliency is a cue that is often used for directing attention of a computer vision system, e.g., in smart environments or for robots. Unlike the majority of recent publications on visual/audio saliency, we aim at a well grounded integration of several modalities. The proposed framework is based on fuzzy aggregations and offers a flexible, plausible, and efficient way for combining multi-modal saliency information. Besides incorporating different modalities, we extend classical 2D saliency maps to multi-camera and multi-modal 3D saliency spaces. For experimental validation we realized the proposed system within a smart environment. The evaluation took place for a demanding setup under real-life conditions, including focus of attention selection for multiple subjects and concurrently active modalities

Saliency-based identification and recognition of pointed-at objects

“…For active scene exploration, saliency can be used to steer the sensors towards salientthus potentially relevant -regions to detect objects of interest (e.g. [8], [10], [12]). Combining these methods, [14] utilized bottom-up attention, stereo vision and SIFT to perform robust and efficient scene analysis on a mobile robot.…”

“…[8]) or audio-visually (e.g. [10], [12]) salient regions is a natural and efficient method to detect and focus interacting persons.…”

Feature Representations for the Recognition of 3D Emblematic Gestures

“…5.1), this is sufficient. Note that, in a multi-camera setting, a view selection algorithm can be applied choosing the two "best" views according to some global criteria [6]. Aligned trajectory points are then projected to 3D by ray casting.…”

“…In particular, we propose representing a gesture by projection on its principal plane of motion, which we call the action plane. For the acquisition of gesture trajectories, we build upon our previous work on 3D pointing gesture recognition [5] and saliency-based view selection in multi-camera setups [6].…”

Sift-Based Camera Localization Using Reference Objects for Application in Multi-Camera Environments and Robotics

“…For this purpose, the identities of the persons in the room have to be determined (see (Salah et al, 2008)) as well as the audio-visual focus of attention has to be estimated (see (Voit and Stiefelhagen, 2010;Schauerte et al, 2009)), e.g. to present personalized information on the display a person is currently looking at.…”