Finding audio-visual events in informal social gatherings

Abstract

In this paper we address the problem of detecting and localizing objects that can be both seen and heard, e.g., people. This may be solved within the framework of data clustering. We propose a new multimodal clustering algorithm based on a Gaussian mixture model, where one of the modalities (visual data) is used to supervise the clustering process. This is made possible by mapping both modalities into the same metric space. To this end, we fully exploit the geometric and physical properties of an audio-visual sensor based on binocular vision and binaural hearing. We propose an EM algorithm that is theoretically well justified, intuitive, and extremely efficient from a computational point of view. This efficiency makes the method implementable on advanced platforms such as humanoid robots. We describe in detail tests and experiments performed with publicly available data sets that yield very interesting results.

Online multimodal speaker detection for humanoid robots

“…That is, the number of speakers as well as their positions and their speaking state. In order to reach this goal, we adopted the framework proposed in [2]. Based on a multimodal Gaussian mixture model (mGMM), this method is able to detect and localize audiovisual events from auditory and visual observations.…”

“…Equation (1) maps 3D points onto the 1D space of ITD observations. The key aspect of our generative audio-visual model [8], [2] is that (1) can be used to map 3D points (visual features) onto the ITD space associated with two microphones, on the premise that the cameras are aligned with the microphones [9]. Hence the fusion between binaural observations and binocular observations is achieved in 1-D.…”

“…Each mixture component is associated with an audio-visual object centered at µ n and with variance σ 2 n . This mixture has the following probability density function: (2) where N is the number of components, i.e., audio-visual objects, π n is the weight of the n th component, N (x ; µ n , σ 2 n ) is the value of the Gaussian distribution at x, U is the value of the uniform distribution accounting for outliers, and Θ = {π n , µ n , σ 2 n }. In this equation, x stands for a realization of the random variable X, shown in the corresponding graphical model on Fig.…”

“…This is done using an ExpectationMaximization (EM) algorithm, derived from the probabilistic graphical model. Notice that with this formulation the number of AV objects N can be estimated from the observed data by maximizing a Bayesian information criterion (BIC) score [2]. However, this implies to run the EM algorithm several times with different values of N , which is prohibitive in the case of an on-line implementation.…”

“…The initial implementations of the nGMM EM algorithm was using 3D points [8], [2] as just described. Alternatively, one can replace 3D points with 3D faces, more precisely with 3D face centers which are fair approximations of 3D mouth positions, i.e., the 3D acoustic emitters.…”