Audiovisual Speaker Tracking Using Nonlinear Dynamical Systems With Dynamic Stream Weights

Schymura, Christopher; Kolossa, Dorothea

doi:10.1109/taslp.2020.2980974

“…( 1) is based on assuming statistical independence between acoustic and visual observations, as well as imposing an uninformative prior on ζi,j ∀ i, j, cf. [16]. However, when working with audiovisual input data, modalities may not be equally informative in all spatial regions due to, e.g., challenging lighting conditions or a directional noise source.…”

Section: Data Fusionmentioning

confidence: 99%

“…Extending this concept to dynamic stream weights (DSWs) allows us to weight the contributions of each input based on their instantaneous reliability. This concept was originally proposed in the context of audiovisual automatic speech recognition (ASR) [14], and has proven valuable for speaker identification [15], but was also recently adopted for speaker localization and tracking [16,17].…”

Section: Introductionmentioning

confidence: 99%

Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain

Wissing

¹

,

Boenninghoff

²

,

Kolossa

³

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Estimating the positions of multiple speakers can be helpful for tasks like automatic speech recognition or speaker diarization. Both applications benefit from a known speaker position when, for instance, applying beamforming or assigning unique speaker identities. Recently, several approaches utilizing acoustic signals augmented with visual data have been proposed for this task. However, both the acoustic and the visual modality may be corrupted in specific spatial regions, for instance due to poor lighting conditions or to the presence of background noise. This paper proposes a novel audiovisual data fusion framework for speaker localization by assigning individual dynamic stream weights to specific regions in the localization space. This fusion is achieved via a neural network, which combines the predictions of individual audio and video trackers based on their time-and location-dependent reliability. A performance evaluation using audiovisual recordings yields promising results, with the proposed fusion approach outperforming all baseline models.

show abstract

“…However, these methods prefer to use the detection results of the single modality to assist the other modality to obtain more accurate observations, while neglecting to fully utilize the complementarity and redundancy of audio-visual information. In addition, most of the existing audio-visual trackers use generation algorithms (Ban et al 2019;Schymura The Thirty-Sixth AAAI Conference on Artificial Intelligence and Kolossa 2020; Qian et al 2017), which are difficult to adapt to random and diverse changes of target appearance. Furthermore, the likelihood calculation based on the color histogram or Euclidean distance is susceptible to interference from observation noise, which limits the performance of the fusion likelihood.…”

Section: Introductionmentioning

confidence: 99%

“…The stGCF-based audio cues are mapped to the localization space consistent with the visual cues. The integrated audio-visual cues combined with perception weights evaluated by the multi-modal perception attention network generate a fusion map that guides update step of the PF-based multi-modal tracker.weight flow(Schymura and Kolossa 2020). Probability Hypothesis Density (PHD) filter is introduced for tracking an unknown and variable number of speakers with the theory of Random Finite Sets (RFSs).…”

mentioning

confidence: 99%

Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking

Li

¹

,

Liu

²

,

Tang

³

2022

AAAI

View full text Add to dashboard Cite

Multi-modal fusion is proven to be an effective method to improve the accuracy and robustness of speaker tracking, especially in complex scenarios. However, how to combine the heterogeneous information and exploit the complementarity of multi-modal signals remains a challenging issue. In this paper, we propose a novel Multi-modal Perception Tracker (MPT) for speaker tracking using both audio and visual modalities. Specifically, a novel acoustic map based on spatial-temporal Global Coherence Field (stGCF) is first constructed for heterogeneous signal fusion, which employs a camera model to map audio cues to the localization space consistent with the visual cues. Then a multi-modal perception attention network is introduced to derive the perception weights that measure the reliability and effectiveness of intermittent audio and video streams disturbed by noise. Moreover, a unique cross-modal self-supervised learning method is presented to model the confidence of audio and visual observations by leveraging the complementarity and consistency between different modalities. Experimental results show that the proposed MPT achieves 98.6% and 78.3% tracking accuracy on the standard and occluded datasets, respectively, which demonstrates its robustness under adverse conditions and outperforms the current state-of-the-art methods.

show abstract

“…However, these methods prefer to use the detection results of the single modality to assist the other modality to obtain more accurate observations, while neglecting to fully utilize the complementarity and redundancy of audio-visual information. In addition, most of the existing audio-visual trackers use generation algorithms (Ban et al 2019;Schymura and Kolossa 2020;Qian et al 2017), which are difficult to adapt to random and diverse changes of target appearance. Furthermore, the likelihood calculation based on the color histogram or Euclidean distance is susceptible to interference from observation noise, which limits the performance of the fusion likelihood.…”

Section: Introductionmentioning

confidence: 99%

Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking

Li¹,

Liu²,

Tang³

2021

Preprint

0

View full text Add to dashboard Cite

Multi-modal fusion is proven to be an effective method to improve the accuracy and robustness of speaker tracking, especially in complex scenarios. However, how to combine the heterogeneous information and exploit the complementarity of multi-modal signals remains a challenging issue. In this paper, we propose a novel Multi-modal Perception Tracker (MPT) for speaker tracking using both audio and visual modalities. Specifically, a novel acoustic map based on spatial-temporal Global Coherence Field (stGCF) is first constructed for heterogeneous signal fusion, which employs a camera model to map audio cues to the localization space consistent with the visual cues. Then a multi-modal perception attention network is introduced to derive the perception weights that measure the reliability and effectiveness of intermittent audio and video streams disturbed by noise. Moreover, a unique cross-modal self-supervised learning method is presented to model the confidence of audio and visual observations by leveraging the complementarity and consistency between different modalities. Experimental results show that the proposed MPT achieves 98.6% and 78.3% tracking accuracy on the standard and occluded datasets, respectively, which demonstrates its robustness under adverse conditions and outperforms the current state-of-the-art methods.

show abstract

Audiovisual Speaker Tracking Using Nonlinear Dynamical Systems With Dynamic Stream Weights

Cited by 7 publications

References 65 publications

Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain

Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain

Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking

Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking

Contact Info

Product

Resources

About