Multimodal saliency-based attention for object-based scene analysis

Abstract

Abstract-Multimodal attention is a key requirement for humanoid robots in order to navigate in complex environments and act as social, cognitive human partners. To this end, robots have to incorporate attention mechanisms that focus the processing on the potentially most relevant stimuli while controlling the sensor orientation to improve the perception of these stimuli. In this paper, we present our implementation of audio-visual saliency-based attention that we integrated in a system for knowledge-driven audio-visual scene analysis and object-based world modeling. For this purpose, we introduce a novel isophote-based method for proto-object segmentation of saliency maps, a surprise-based auditory saliency definition, and a parametric 3-D model for multimodal saliency fusion. The applicability of the proposed system is demonstrated in a series of experiments.Index Terms-audio-visual saliency, auditory surprise, isophote-based visual proto-objects, parametric 3-D saliency model, object-based inhibition of return, multimodal attention, scene exploration, hierarchical object analysis, overt attention, active perception

Quaternion-Based Spectral Saliency Detection for Eye Fixation Prediction

“…Moreover, predicting where humans look is not only an interesting question in cognitive psychology, neurophysics, and neurobiology, but it has proven to be an important information for many application areas, e.g. : for ecient scene exploration and analysis in robotics (see, e.g., [7,8]), information visualization using image retargeting (see, e.g., [9,10]), or predicting the attractiveness of advertisement (see [11]). …”

Multimodal saliency-based attention: A lazy robot's approach

“…kristian.kroschel@iosb-extern.fraunhofer. de We extend our previous work described in [2], which introduced a multimodal attention system for object-based audio-visual scene exploration and analysis. In this contribution, we integrated adaptive predictive gaze control [3] to control a kinematically redundant robot head, i.e.…”

“…Due to the practical relevance of attention for autonomous robots in complex natural environments, computational attention models have attracted an increasing interest in the field of robotics during the last decade (e. g., [2], [5]- [11]). Accordingly, there exists a wide range of different attention models and the selection of the saliency definition heavily influences which signal components attract the attention.…”

“…In coherence theory of visual cognition, proto-objects refer to volatile units of information that can be accessed by selective attention and later validated as actual object [16]. However, it is difficult to determine the image region that approximates the extent of a (proto-)object at the attended location (see [16] the saliency map in order to determine salient regions and roughly approximate the extent of the proto-object region [2]. In contrast to visual saliency, few models to determine salient auditory signals have been proposed that are suitable for robotics applications (e.g., [2], [17], [18]).…”

“…However, it is difficult to determine the image region that approximates the extent of a (proto-)object at the attended location (see [16] the saliency map in order to determine salient regions and roughly approximate the extent of the proto-object region [2]. In contrast to visual saliency, few models to determine salient auditory signals have been proposed that are suitable for robotics applications (e.g., [2], [17], [18]). To this end, we introduced auditory surprise in [2], which is based on the accumulated Bayesian surprise [19] of all frequencies.…”