Multimodal (audio–visual) source separation exploiting multi-speaker tracking, robust beamforming and time–frequency masking

Abstract

A novel multimodal source separation approach is proposed for physically moving and stationary sources which exploits a circular microphone array, multiple video cameras, robust spatial beamforming and time-frequency masking. The challenge of separating moving sources, including higher reverberation time (RT) even for physically stationary sources, is that the mixing filters are time varying; as such the unmixing filters should also be time varying but these are difficult to determine from only audio measurements. Therefore in the proposed approach, visual modality is used to facilitate the separation for both stationary and moving sources. The movement of the sources is detected by a three-dimensional tracker based on a Markov Chain Monte Carlo particle filter. The audio separation is performed by a robust least squares frequency invariant data-independent beamformer. The uncertainties in source localisation and direction of arrival information obtained from the 3D video-based tracker are controlled by using a convex optimisation approach in the beamformer design. In the final stage, the separated audio sources are further enhanced by applying a binary time-frequency masking technique in the cepstral domain. Experimental results show that using the visual modality, the proposed algorithm cannot only achieve performance better than conventional frequency-domain source separations algorithms, but also provide acceptable separation performance for moving sources.

Automatic adaptive speech separation using beamformer-output-ratio for voice activity classification

“…In general, ABS-IPC systems can enhance the performance by using larger false alarm rates. For example, in test EPas2.4, when the false alarm rate is increased to 0.1, the output SINRs of the ABS-IPC system increase to (12,15) dB (while the output SINRs of the ABS system drop to (5, 13) dB).…”

“…Other existing approaches for active speaker identification in multi-speaker environments include video signal processing, speaker recognition, and multi-source localisation techniques aided by clustering using Gaussian mixture models and expectation maximisation [15][16][17][18]. Another approach that is related to the VAC problem is to detect the number of sources.…”

Exploiting the Complementarity of Audio and Visual Data in Multi-speaker Tracking

“…Still in a particle filter tracking framework, [8] proposed to use the maximum global coherence field of the audio signal and image colorhistogram matching to adapt the reliability of audio and visual information. Finally, along a different line, [9] used visual tracking information to assist source separation and beamforming.…”