Three-Dimensional Speaker Localization: Audio-Refined Visual Scaling Factor Estimation

Abstract

Neither a monocular RGB camera nor a smallsize microphone array is capable of accurate three-dimensional (3D) speaker localization. By taking advantage of accurate visual object detection, and audio-visual complementary sensor fusion, we formulate the three-dimensional (3D) speaker localization problem as a visual scaling factor estimation problem. As a result, we effectively reduce the traditional audio-only 3D speaker localization from an exhaustive grid search to a one-dimensional (1D) optimization problem. We propose a multi-modal perception system with two optimization approaches. We show that the proposed methods are effective, accurate, and robust against interference and, as corroborated by indicative empirical results on real dataset, competitive to the conventional uni-modal and the state-of-the-art audio-visual speaker localization approaches.

Audiovisual Tracking of Multiple Speakers in Smart Spaces

“…Given the increased availability of easy-to-deploy audio and video sensors and the improvements in computing facilities, in recent years, there has been a relevant growth in the number of proposals for multiple speakers tracking (Multiple-Object Tracking, MOT) in smart spaces, combining audio and video information [ 17 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 ]. Qian et al in [ 17 ] conducted an extensive review of state of the art in audiovisual speaker tracking.…”

“…The most commonly used approach is based on an observation model that keeps the Bayesian scheme. There are many examples of face detectors based on deep learning [ 25 , 26 , 27 , 28 , 29 , 31 ], although Siamese networks have also been used to generate measures of particle similarity to previous reference images of each target [ 32 , 39 ], and fusion models based on the attention mechanism [ 32 ]. Fewer proposals we found with end-to-end trained audiovisual solutions as in [ 24 , 40 ] for object tracking in which visual and auditory inputs are fused by an added fusion layer.…”