Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers

Ban, Yutong; Alameda-Pineda, Xavier; Girin, Laurent; Horaud, Radu

doi:10.1109/tpami.2019.2953020

Cited by 43 publications

(49 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A diarization system using only face identification and SSL may be regarded 5 Note that SA-WER used here is different from SWER of [2]. as a baseline as this approach was widely used in previous audiovisual diarization studies [34][35][36]. The results show that the use of speaker identification substantially improved the speaker attribution accuracy.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Advances in Online Audio-Visual Meeting Transcription

Yoshioka

Huang

Hurvitz

et al. 2019

2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

View full text Add to dashboard Cite

This paper describes a system that generates speaker-annotated transcripts of meetings by using a microphone array and a 360-degree camera. The hallmark of the system is its ability to handle overlapped speech, which has been an unsolved problem in realistic settings for over a decade. We show that this problem can be addressed by using a continuous speech separation approach. In addition, we describe an online audio-visual speaker diarization method that leverages face tracking and identification, sound source localization, speaker identification, and, if available, prior speaker information for robustness to various real world challenges. All components are integrated in a meeting transcription framework called SRD, which stands for "separate, recognize, and diarize". Experimental results using recordings of natural meetings involving up to 11 attendees are reported. The continuous speech separation improves a word error rate (WER) by 16.1% compared with a highly tuned beamformer. When a complete list of meeting attendees is available, the discrepancy between WER and speaker-attributed WER is only 1.0%, indicating accurate wordto-speaker association. This increases marginally to 1.6% when 50% of the attendees are unknown to the system.

show abstract

Section: Resultsmentioning

confidence: 99%

“…[33,34] do not cope with speech overlaps. While the methods proposed in [35,36] address the overlap issue, they rely solely on spatial cues and thus are not applicable when multiple speakers sit side by side.…”

Section: Introductionmentioning

confidence: 99%

Advances in Online Audio-Visual Meeting Transcription

Yoshioka

Huang

Hurvitz

et al. 2019

2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

View full text Add to dashboard Cite

show abstract

“…The present work is also related to previous work in localizing sounds in visual inputs [20,14,22,9,8,24,4,35], which aims to identify which pixels in a video are associated with an object making a particular sound.…”

Section: Sound Localizationmentioning

confidence: 97%

Self-Supervised Moving Vehicle Tracking With Stereo Sound

Gan

Zhao²,

Chen³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

164

102

View full text Add to dashboard Cite

Humans are able to localize objects in the environment using both visual and auditory cues, integrating information from multiple modalities into a common reference frame. We introduce a system that can leverage unlabeled audiovisual data to learn to localize objects (moving vehicles) in a visual reference frame, purely using stereo sound at inference time. Since it is labor-intensive to manually annotate the correspondences between audio and object bounding boxes, we achieve this goal by using the co-occurrence of visual and audio streams in unlabeled videos as a form of selfsupervision, without resorting to the collection of ground truth annotations. In particular, we propose a framework that consists of a vision "teacher" network and a stereosound "student" network. During training, knowledge embodied in a well-established visual vehicle detection model is transferred to the audio domain using unlabeled videos as a bridge. At test time, the stereo-sound student network can work independently to perform object localization using just stereo audio and camera meta-data, without any visual input. Experimental results on a newly collected Auditory Vehicle Tracking dataset verify that our proposed approach outperforms several baseline approaches. We also demonstrate that our cross-modal auditory localization approach can assist in the visual localization of moving vehicles under poor lighting conditions.

show abstract

“…In this section we derive the expression for τ j by computing the integral (17). Using the probabilistic model defined, we can write (the index j is omitted): We will first marginalize s t−L .…”

Section: Appendix D Derivation Of the Birth Probabilitymentioning

confidence: 99%

Tracking Multiple Audio Sources With the von Mises Distribution and Variational EM

Ban

Alameda-Pineda

Evers

et al. 2019

IEEE Signal Process. Lett.

Self Cite

View full text Add to dashboard Cite

In this paper we address the problem of simultaneously tracking several moving audio sources, namely the problem of estimating source trajectories from a sequence of observed features. We propose to use the von Mises distribution to model audio-source directions of arrival with circular random variables. This leads to a Bayesian filtering formulation which is intractable because of the combinatorial explosion of associating observed variables with latent variables, over time. We propose a variational approximation of the filtering distribution. We infer a variational expectation-maximization algorithm that is both computationally tractable and time efficient. We propose an audio-source birth method that favors smooth source trajectories and which is used both to initialize the number of active sources and to detect new sources. We perform experiments with the recently released LOCATA dataset comprising two moving sources and a moving microphone array mounted onto a robot.

show abstract

Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers

Cited by 43 publications

References 38 publications

Advances in Online Audio-Visual Meeting Transcription

Advances in Online Audio-Visual Meeting Transcription

Self-Supervised Moving Vehicle Tracking With Stereo Sound

Tracking Multiple Audio Sources With the von Mises Distribution and Variational EM

Contact Info

Product

Resources

About