Audio constrained particle filter based visual tracking

Kılıç, Volkan; Barnard, Mark; Wang, Wenwu; Kittler, Josef

doi:10.1109/icassp.2013.6638334

Cited by 11 publications

(17 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The DOA data is introduced to the SMC-PHD filter based on [34] and [36] where the efficiency of the particles is improved under a particle filter framework by re-allocating all the particles around the DOA line which is drawn from the center of the microphone array to a point in the image frame estimated by the projection of DOA to 2D image plane. However, different from [34] and [36] in which the DOA is used in the same way for all the particles, here the contribution of the DOA information is varied depending on the type of the particles.…”

Section: Audio-visual Tracker With Smc-phd Filtermentioning

confidence: 99%

“…However, different from [34] and [36] in which the DOA is used in the same way for all the particles, here the contribution of the DOA information is varied depending on the type of the particles. Similar to [34] and [36], we also use the samspare-mean (SSM) method [48] for the DOA estimation which is further enhanced by a third-order Auto-Regressive (AR) model. We should note that there are other audio features and algorithms for extracting these features that could be used in our proposed system, however, exploring other audio detection methods is beyond the scope of this work.…”

Section: Audio-visual Tracker With Smc-phd Filtermentioning

confidence: 99%

“…If the DOA is available in current frame, the DOA line is drawn [34] and the perpendicular Euclidean distances…”

Section: Audio-visual Tracker With Smc-phd Filtermentioning

confidence: 99%

“…More specifically, the propagation of the born particles is decided based on the DOA information and the particles are re-located around the line drawn upon the DOA. A similar approach has been used in [34], [35] and [36] under the PF framework for a fixed number of speakers. Here, the SMC-PHD filter is used, and to our knowledge, audio information has not been previously used with visual information in a SMC-PHD filter as we do here.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Mean-Shift and Sparse Sampling-Based SMC-PHD Filtering for Audio Informed Visual Speaker Tracking

Kılıç

Barnard

Wang

et al. 2016

IEEE Trans. Multimedia

Self Cite

View full text Add to dashboard Cite

Abstract-The probability hypothesis density (PHD) filter based on sequential Monte Carlo (SMC) approximation (also known as SMC-PHD filter) has proven to be a promising algorithm for multi-speaker tracking. However, it has a heavy computational cost as surviving, spawned and born particles need to be distributed in each frame to model the state of the speakers and to estimate jointly the variable number of speakers with their states. In particular, the computational cost is mostly caused by the born particles as they need to be propagated over the entire image in every frame to detect the new speaker presence in the view of the visual tracker. In this paper, we propose to use audio data to improve the visual SMC-PHD (V-SMC-PHD) filter by using the direction of arrival (DOA) angles of the audio sources to determine when to propagate the born particles and re-allocate the surviving and spawned particles. The tracking accuracy of the AV-SMC-PHD algorithm is further improved by using a modified mean-shift algorithm to search and climb density gradients iteratively to find the peak of the probability distribution, and the extra computational complexity introduced by mean-shift is controlled with a sparse sampling technique. These improved algorithms, named as AVMS-SMC-PHD and sparse-AVMS-SMC-PHD respectively, are compared systematically with AV-SMC-PHD and V-SMC-PHD based on the AV16.3, AMI and CLEAR datasets.

show abstract

Section: Audio-visual Tracker With Smc-phd Filtermentioning

confidence: 99%

Section: Audio-visual Tracker With Smc-phd Filtermentioning

confidence: 99%

“…If the DOA is available in current frame, the DOA line is drawn [34] and the perpendicular Euclidean distances…”

Section: Audio-visual Tracker With Smc-phd Filtermentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Mean-Shift and Sparse Sampling-Based SMC-PHD Filtering for Audio Informed Visual Speaker Tracking

Kılıç

Barnard

Wang

et al. 2016

IEEE Trans. Multimedia

Self Cite

View full text Add to dashboard Cite

show abstract

“…If the calibration information is not available, the positions of the microphone arrays could be estimated via microphone self-calibration [50] or combined microphone and camera calibrations [51], which is however beyond the scope of this study. The idea behind our approach is to relocate the distributed particles around the DOA line and then re-calculate the weights of the relocated particles according to their distance to the DOA line [52]. The DOA line can be drawn as follows.…”

Section: Proposed Particle Filter Based Audio Constraint Visual mentioning

confidence: 99%

Audio Assisted Robust Visual Tracking With Adaptive Particle Filtering

Kılıç

Barnard

Wang

et al. 2015

IEEE Trans. Multimedia

Self Cite

View full text Add to dashboard Cite

Abstract-The problem of tracking multiple moving speakers in indoor environments has recently received much attention. Earlier techniques were based purely on vision, but the theoretical and algorithmic advances and a constant growth in speed of processing have led to the emergence of techniques which allow the fusion of audio and visual data. The fusion of multi-modal information has been shown to be instrumental in improving tracking performance, as well as robustness in the case of challenging situations like occlusions (by the limited field of view of cameras or by other speakers). However, data fusion algorithms often suffer from noise corrupting the sensor measurements which cause non-negligible detection errors. Here, a novel approach to combining audio and visual data is proposed. In our framework, we employ audio data as an aid to particle filter (PF) based visual tracking, by using the direction of arrival angles of the audio sources to reshape the typical Gaussian noise distribution of particles in the propagation step and to weight the observation model in the measurement step. This approach is further improved by solving a typical problem associated with the PF. It has been observed that the efficiency and accuracy of the PF usually depend on the number of particles and noise variance used in the estimation and propagation functions for re-allocating these particles at each iteration. Both of these parameters are specified beforehand and are kept fixed in the regular implementation of the PF which makes the tracker unstable in practice. To address these problems, we design an algorithm which adapts both the number of particles and noise variance based on tracking error and the area occupied by the particles in the image. Experiments on the AV 16.3 dataset show the advantage of our proposed methods over the baseline PF method and an existing adaptive PF algorithm for tracking occluded speakers with a significantly reduced number of particles.Index Terms-Audio-visual speaker tracking, particle filter, adaptive particle filter.

show abstract

Robust audiovisual speech recognition using noise-adaptive linear discriminant analysis

Zeiler

Nicheli

et al. 2016

2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Automatic speech recognition (ASR) has become a widespread and convenient mode of human-machine interaction, but it is still not sufficiently reliable when used under highly noisy or reverberant conditions. One option for achieving far greater robustness is to include another modality that is unaffected by acoustic noise, such as video information. Currently the most successful approaches for such audiovisual ASR systems, coupled hidden Markov models (HMMs) and turbo decoding, both allow for slight asynchrony between audio and video features, and significantly improve recognition rates in this way. However, both typically still neglect residual errors in the estimation of audio features, so-called observation uncertainties. This paper compares two strategies for adding these observation uncertainties into the decoder, and shows that significant recognition rate improvements are achievable for both coupled HMMs and turbo decoding.

show abstract

Audio constrained particle filter based visual tracking

Cited by 11 publications

References 12 publications

Mean-Shift and Sparse Sampling-Based SMC-PHD Filtering for Audio Informed Visual Speaker Tracking

Mean-Shift and Sparse Sampling-Based SMC-PHD Filtering for Audio Informed Visual Speaker Tracking

Audio Assisted Robust Visual Tracking With Adaptive Particle Filtering

Robust audiovisual speech recognition using noise-adaptive linear discriminant analysis

Contact Info

Product

Resources

About