An EM algorithm for joint source separation and diarisation of multichannel convolutive speech mixtures

Abstract

We present a probabilistic model for joint source separation and diarisation of multichannel convolutive speech mixtures. We build upon the framework of local Gaussian model (LGM) with non-negative matrix factorization (NMF). The diarisation is introduced as a temporal labeling of each source in the mix as active or inactive at the short-term frame level. We devise an EM algorithm in which the source separation process is aided by the diarisation state, since the latter indicates the sources actually present in the mixture. The diarisation state is tracked with a Hidden Markov Model (HMM) with emission probabilities calculated from the estimated source signals. The proposed EM has separation performance comparable with a state-of-the-art LGM NMF method, while outperforming a state-of-the-art speaker diarisation pipeline.

Exploiting the intermittency of speech for joint separation and diarization

“…In the present paper we extend the work in [14] by proposing a probabilistic model for simultaneous MASS and diarization of under-determined convolutive speech mixtures, that does not use the narrow-band assumption but is based instead on the spatial covariance matrix model (SCM) [1,16]. The source activity model we use here is a variant of [14].…”

“…A categorical random variable Z = n, n ∈ [1, N ] selects the diarization dZ at frame . Z is modelled by an HMM as in [14]:…”

“…The use of diarization has been used for source tracking in [12] and for Support from EU-FP7 ERC Advanced Grant VHIA (#340113) is greatly acknowledged. source separation in [13,14]. In [13] the tested mixtures have more microphones than the number of sources.…”

“…The source activity model we use here is a variant of [14]. In contrast to [10] and [11], we now process the activity of all sources jointly as a diarization state, in order to exploit the potential correlations on speaker activity.…”

“…In [13] the tested mixtures have more microphones than the number of sources. In [14] the convolutive mixing is based on the narrow-band assumption [15] which is apt only in low reverberation.…”