Multimodal saliency-based attention for object-based scene analysis

Abstract

Abstract-Multimodal attention is a key requirement for humanoid robots in order to navigate in complex environments and act as social, cognitive human partners. To this end, robots have to incorporate attention mechanisms that focus the processing on the potentially most relevant stimuli while controlling the sensor orientation to improve the perception of these stimuli. In this paper, we present our implementation of audio-visual saliency-based attention that we integrated in a system for knowledge-driven audio-visual scene analysis and object-based world modeling. For this purpose, we introduce a novel isophote-based method for proto-object segmentation of saliency maps, a surprise-based auditory saliency definition, and a parametric 3-D model for multimodal saliency fusion. The applicability of the proposed system is demonstrated in a series of experiments.Index Terms-audio-visual saliency, auditory surprise, isophote-based visual proto-objects, parametric 3-D saliency model, object-based inhibition of return, multimodal attention, scene exploration, hierarchical object analysis, overt attention, active perception

“Wow!” Bayesian surprise for salient acoustic event detection

“…In recent years computational models of attention have attracted an increasing interest in the field of robotics (see, e.g., [1,2]) and various other application areas (see, e.g., [3, Fig. 1.…”

“…1. Our robotic target platform: The Karlsruhe Humanoid Head and the ARMAR humanoid robot (see [18]), which served as platform for our qualitative system evaluations [1]. [6][7][8][9]).…”

“…to highlight potentially relevant and thus interesting -that is to say "salient" -data, the vagueness and task-dependence of this problem description leads to a variety of models that may differ substantially in which parts of the signal they mark as being of interest. Unfortunately, in contrast to the fast growing amount of proposed visual saliency models (see [10,11]), only few practically applicable models for acoustic attention exist (e.g., [1,6,7]). Most closely related to our work is the model described by Kayser et al [6] which is based on the well-known visual saliency model of Itti et al [12] and, most notably, has been successfully applied to speech processing by Kalinli et al [7] and, in principle, by Lin et al [8] to allow for faster human acoustic event detection through audio visualization.…”

“…Unfortunately, auditory and acoustic attention has not yet found its way into as many practical applications as its visual counterpart (see, e.g., [1][2][3]). One of the possible reasons for this could be that only few run-time efficient, reliable,…”

“…We primarily developed acoustic surprise to focus the computational resources and control the overt attention of (humanoid) robots, see Fig. 1, and smart environments (see [1,[3][4][5]). In such applications, acoustic surprise can serve two purposes: First, we can focus audio processing and, second, we can actively control the overt attention (i.e., the sensor orientation) to optimize the scene perception.…”