Vision-guided robot hearing

Abstract

Natural human-robot interaction in complex and unpredictable environments is one of the main research lines in robotics. In typical real-world scenarios, humans are at some distance from the robot and the acquired signals are strongly impaired by noise, reverberations and other interfering sources. In this context, the detection and localisation of speakers plays a key role since it is the pillar on which several tasks (e.g.: speech recognition and speaker tracking) rely. We address the problem of how to detect and localize people that are both seen and heard by a humanoid robot. We introduce a hybrid deterministic/probabilistic model. Indeed, the deterministic component allows us to map the visual information into the auditory space. By means of the probabilistic component, the visual features guide the grouping of the auditory features in order to form AV objects. The proposed model and the associated algorithm are implemented in real-time (17 FPS) using a stereoscopic camera pair and two microphones embedded into the head of the humanoid robot NAO. We performed experiments on (i) synthetic data, (ii) a publicly available data set and (iii) data acquired using the robot. The results we obtained validate the approach and encourage us to further investigate how vision can help robot hearing.

Extending the Cascaded Gaussian Mixture Regression Framework for Cross-Speaker Acoustic-Articulatory Mapping

“…Setting p(z|x, y, m) = p(z|x, m) is equivalent to say that Z and Y are conditionally independent given x, which can be expressed equivalently . We now further calculate this expression for the J-GMM model defined in (7). Injecting 7into (9) leads to:…”

“…In contrast, we provided detailed derivation in [18], where (19) is shown to result into two equivalent forms of a GMR expression (25) and (26). Also, to be fully precise, (7) in [21] corresponds to (26) in [18] up to two differences that we interpret as typos: First, the term Σ…”

“…voice conversion [2], [3], in image processing, e.g. head pose estimation from depth data [4], generation of hand writing [5], and in robotics [6], [7]. In the present paper, we consider the application of GMR to the speech acoustic-articulatory inversion problem, i.e.…”

Exploiting the Complementarity of Audio and Visual Data in Multi-speaker Tracking

“…In this paper we propose a novel multi-speaker tracking method inspired from previous research on "instantaneous" audio-visual fusion [11,12]. A dynamic Bayesian model is investigated to smoothly fuse acoustic and visual information over time from their feature spaces.…”

Tracking the Active Speaker Based on a Joint Audio-Visual Observation Model

“…In this paper we propose to enforce audio-visual spatial coincidence, e.g., [1,8,10], rather than temporal coincidence, e.g., correlation [9,16], into diarization. We consider a setup consisting of people that are engaged in a multiparty conversation while they are free to move and to turn their attention away from the cameras.…”