Active-speaker detection and localization with microphones and cameras embedded into a robotic head

Abstract

Abstract-In this paper we present a method for detecting and localizing an active speaker, i.e., a speaker that emits a sound, through the fusion between visual reconstruction with a stereoscopic camera pair and sound-source localization with several microphones. Both the cameras and the microphones are embedded into the head of a humanoid robot. The proposed statistical fusion model associates 3D faces of potential speakers with 2D sound directions. The paper has two contributions: (i) a method that discretizes the two-dimensional space of all possible sound directions and that accumulates evidence for each direction by estimating the time difference of arrival (TDOA) over all the microphone pairs, such that all the microphones are used simultaneously and symmetrically and (ii) an audio-visual alignment method that maps 3D visual features onto 2D sound directions and onto TDOAs between microphone pairs. This allows to implicitly represent both sensing modalities into a common audiovisual coordinate frame. Using simulated as well as real data, we quantitatively assess the robustness of the method against noise and reverberations, and we compare it with several other methods. Finally, we describe a realtime implementation using the proposed technique and with a humanoid head embedding four microphones and two cameras: this enables natural human-robot interactive behavior.

Synthetic Aperture Computation as the Head is Turned in Binaural Direction Finding

“…In the absence of visual cues, for example in complete darkness, we tend to turn our heads in a quite exaggerated way to explore aural signal. This suggests that visual and aural directional cues are integrated by a top level worldview manager incorporating information from both (plus other) types of sensory system, and this approach has been exploited in robotic systems (e.g., [35]). …”

“…It is only relatively recently that binaural sensing in robotic systems has developed sufficiently for the deployment of processes such as finding directions to acoustic sources [25][26][27][28][29][30][31][32][33][34][35][36]. Acoustic source localization has been largely restricted to estimating azimuth [26][27][28][29][30][31][32][33] on the assumption of zero elevation, except where audition has been fused with vision for estimates also of elevation [34,35,37,38].…”

“…Acoustic source localization has been largely restricted to estimating azimuth [26][27][28][29][30][31][32][33] on the assumption of zero elevation, except where audition has been fused with vision for estimates also of elevation [34,35,37,38]. Information gathered as the head is turned has been exploited either to locate the azimuth at which ITD reduces to zero thereby determining the azimuthal direction to a source, or to resolve the front-back ambiguity associated with estimating only azimuth [28][29][30][31][32][33][34]39,40].…”

Application of Short Time Fourier Transform and Wavelet Transform for Sound Source Localization Using Single Moving Microphone in Machine Condition Monitoring

“…Then, to localize sound sources using acoustic signals, some methods also have been developed. For instance, sound source localization using time frequency histogram by two microphones [6], through the fusion between visual reconstruction with a stereoscopic camera pair with several microphones [7], and the application of envelope and wavelet transform to enhance the resolution of the received signals through the combination of different time-frequency contents [8].…”

Binaural Range Finding from Synthetic Aperture Computation as the Head is Turned

“…Acoustic source localization has been largely restricted to estimating azimuth [1-13] on the assumption of zero elevation, except where audition has been fused with vision for estimates also of elevation [14,15,17,18]. Information gathered as the head is turned has been exploited either to locate the azimuth at which the ITD reduces to zero, thereby determining the azimuthal direction to a source, or to resolve the front-back ambiguity associated with estimating only the azimuth [4][5][6][7][8][9][10][11][12][13][14]19,20].…”