Audio-Visual Based Online Multi-Source Separation

Ong, Jonah; Vo, Ba-Tuong; Nordholm, Sven; Vo, Ba Ngu; Moratuwage, Diluka; Shim, Changbeom

doi:10.1109/taslp.2022.3156758

Cited by 7 publications

(7 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ong et al [23] designed a cutting-edge method for real-time online multi-source separation that combines audio and visual data to determine the speakers' locations and separate the intended speech from background noise. They developed a deep neural network that integrates both visual and auditory features to carry out this processing.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Audio-Visual Source Separation Based Fusion Techniques

Mudhafar,

Al Tmeme

2024

RIA

View full text Add to dashboard Cite

A novel hybrid deep learning model for audio-visual source separation is introduced in this paper, with a specific focus on the precise isolation of a particular speaker's voice from video content. By leveraging both audio and visual characteristics, the achievement of accurate separation of the targeted speech signal is facilitated by our model. Notably, the incorporation of the speaker's facial expressions as an auxiliary cue for enhancing the extraction of their unique vocal qualities is emphasized. Proficiency in audio-visual speech separation and latent representations of distinctive speaker attributes, known as speaker embeddings, is simultaneously acquired by our model through unsupervised learning on unannotated video data. The model employed in this study is speaker-independent, wherein an initial stage of feature extraction is conducted for both audio and visual inputs prior to the subsequent deep modal analysis. The utilization of facial attribute features as an identifying code enables the identification of the speaker's frequency space or other audio properties. The model's efficacy was assessed through evaluation on the widely recognized AVspeech dataset yielding an improvement of 7.7 in terms of source-to-distortion ratio (SDR).

show abstract

Section: Related Workmentioning

confidence: 99%

“…Ong et al [23] Real-time online multi-source separation that estimates speaker position and separates target speech from background noise using audio and visual data.…”

Section: The Efficacy Of Noise and Reverberation Removalmentioning

confidence: 99%

Audio-Visual Source Separation Based Fusion Techniques

Mudhafar,

Al Tmeme

2024

RIA

View full text Add to dashboard Cite

show abstract

“…The target state is represented as Bernoulli RFS, which is empty or has a single element. In [1], generalized labeled Bernoulli filter (GLMB) is employed to solve the problem of multi-modal space-time permutation and deal with the problem of varying number of speakers. In [21], the Poisson multi-Bernoulli mixture (PMBM) filter is proposed for multi-target speaker tracking, which employs Poisson distributions to represent undetected targets and employs a multi-Bernoulli mixture to represent detected targets with different data association strategies.…”

Section: A Speaker Trackingmentioning

confidence: 99%

“…S PEAKER tracking plays an important role in speech separation [1], speech enhancement [2] and speaker diarization [3]. The task of speaker tracking is to estimate the 2D position, 3D position or Direction of Arrival (DOA) of speakers at each time step.…”

Section: Introductionmentioning

confidence: 99%

Attention-Based End-to-End Differentiable Particle Filter for Audio Speaker Tracking

Zhao,

Xu,

Qian

et al. 2024

IEEE Open J. Signal Process.

View full text Add to dashboard Cite

Particle filters (PFs) have been widely used in speaker tracking due to their capability in modeling a non-linear process or a non-Gaussian environment. However, particle filters are limited by several issues. For example, pre-defined handcrafted measurements are often used which can limit the model performance. In addition, the transition and update models are often preset which make PF less flexible to be adapted to different scenarios. To address these issues, we propose an end-to-end differentiable particle filter framework by employing the multi-head attention to model the long-range dependencies. The proposed model employs the self-attention as the learned transition model and the cross-attention as the learned update model. To our knowledge, this is the first proposal of combining particle filter and transformer for speaker tracking, where the measurement extraction, transition and update steps are integrated into an end-to-end architecture. Experimental results show that the proposed model achieves superior performance over the recurrent baseline models.INDEX TERMS Particle filter, differentiable particle filter, transformer, end to end training, speaker tracking.

show abstract

“…For instance, in [5], spatial search was used to decompose the filtering density into independent GLMBs that are processed in parallel, while, in [21], a parallel centralized implementation for multiple sensors was developed. The GLMB filter has also found a host of applications from robotics [22], sensor networks [23], [24], cell biology [14], [25] to audio/video processing [15], [26].…”

mentioning

confidence: 99%

Linear Complexity Gibbs Sampling for Generalized Labeled Multi-Bernoulli Filtering

Shim

et al. 2023

IEEE Trans. Signal Process.

Self Cite

View full text Add to dashboard Cite

Generalized Labeled Multi-Bernoulli (GLMB) densities arise in a host of multi-object system applications analogous to Gaussians in single-object filtering. However, computing the GLMB filtering density requires solving NP-hard problems. To alleviate this computational bottleneck, we develop a linear complexity Gibbs sampling framework for GLMB density computation. Specifically, we propose a tempered Gibbs sampler that exploits the structure of the GLMB filtering density to achieve an O(T (P + M )) complexity, where T is the number of iterations of the algorithm, P and M are the number hypothesized objects and measurements. This innovation enables the GLMB filter implementation to be reduced from an O(T P 2 M ) complexity to O(T (P + M + log T ) + P M). Moreover, the proposed framework provides the flexibility for trade-offs between tracking performance and computational load. Convergence of the proposed Gibbs sampler is established, and numerical studies are presented to validate the proposed GLMB filter implementation.

show abstract

Audio-Visual Based Online Multi-Source Separation

Cited by 7 publications

References 61 publications

Audio-Visual Source Separation Based Fusion Techniques

Audio-Visual Source Separation Based Fusion Techniques

Attention-Based End-to-End Differentiable Particle Filter for Audio Speaker Tracking

Linear Complexity Gibbs Sampling for Generalized Labeled Multi-Bernoulli Filtering

Contact Info

Product

Resources

About