Audio Assisted Robust Visual Tracking With Adaptive Particle Filtering

Abstract

Abstract-The problem of tracking multiple moving speakers in indoor environments has recently received much attention. Earlier techniques were based purely on vision, but the theoretical and algorithmic advances and a constant growth in speed of processing have led to the emergence of techniques which allow the fusion of audio and visual data. The fusion of multi-modal information has been shown to be instrumental in improving tracking performance, as well as robustness in the case of challenging situations like occlusions (by the limited field of view of cameras or by other speakers). However, data fusion algorithms often suffer from noise corrupting the sensor measurements which cause non-negligible detection errors. Here, a novel approach to combining audio and visual data is proposed. In our framework, we employ audio data as an aid to particle filter (PF) based visual tracking, by using the direction of arrival angles of the audio sources to reshape the typical Gaussian noise distribution of particles in the propagation step and to weight the observation model in the measurement step. This approach is further improved by solving a typical problem associated with the PF. It has been observed that the efficiency and accuracy of the PF usually depend on the number of particles and noise variance used in the estimation and propagation functions for re-allocating these particles at each iteration. Both of these parameters are specified beforehand and are kept fixed in the regular implementation of the PF which makes the tracker unstable in practice. To address these problems, we design an algorithm which adapts both the number of particles and noise variance based on tracking error and the area occupied by the particles in the image. Experiments on the AV 16.3 dataset show the advantage of our proposed methods over the baseline PF method and an existing adaptive PF algorithm for tracking occluded speakers with a significantly reduced number of particles.Index Terms-Audio-visual speaker tracking, particle filter, adaptive particle filter.

Mean-Shift and Sparse Sampling-Based SMC-PHD Filtering for Audio Informed Visual Speaker Tracking

“…For the evolution of the time dependent speaker state, the constant velocity model is employed [36], [45] given as,…”

“…The DOA data is introduced to the SMC-PHD filter based on [34] and [36] where the efficiency of the particles is improved under a particle filter framework by re-allocating all the particles around the DOA line which is drawn from the center of the microphone array to a point in the image frame estimated by the projection of DOA to 2D image plane. However, different from [34] and [36] in which the DOA is used in the same way for all the particles, here the contribution of the DOA information is varied depending on the type of the particles.…”

“…However, different from [34] and [36] in which the DOA is used in the same way for all the particles, here the contribution of the DOA information is varied depending on the type of the particles. Similar to [34] and [36], we also use the samspare-mean (SSM) method [48] for the DOA estimation which is further enhanced by a third-order Auto-Regressive (AR) model. We should note that there are other audio features and algorithms for extracting these features that could be used in our proposed system, however, exploring other audio detection methods is beyond the scope of this work.…”

“…More specifically, the propagation of the born particles is decided based on the DOA information and the particles are re-located around the line drawn upon the DOA. A similar approach has been used in [34], [35] and [36] under the PF framework for a fixed number of speakers. Here, the SMC-PHD filter is used, and to our knowledge, audio information has not been previously used with visual information in a SMC-PHD filter as we do here.…”

Person Tracking Using Audio and Depth Cues

“…There is a consensus that different modalities are complementary to each other, which has motivated an increasing interest in cross-modal tracking in the last decade. Most of these works are done in the audio-visual domain [24,8,9]. Combination of other modalities has recently started to become more popular.…”