Combination of deep speaker embeddings for diarisation

Sun, Guangzhi; Zhang, Chao; Woodland, Philip C.

doi:10.1016/j.neunet.2021.04.020

Cited by 11 publications

(8 citation statements)

References 91 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A pooling layer is often used to convert the frame-level speaker representations to a fixed-length vector for several seconds of audio. Commonly used pooling layers include the means and standard deviations [24] and those with attention mechanisms [1,25] Rather than using the normal cross entropy loss with a softmax activation, some additional loss functions have been proposed to better discriminate between different speaker classes, such as the angular softmax loss [26]. The additive angular margin loss [27] is used for all SC training in this paper.…”

Section: Sc For Speaker Embedding Extraction and Clusteringmentioning

confidence: 99%

“…The data preparation of the AMI corpus for ASR follows the pipeline provided in ESPnet [31]. The same references for speaker diarisation from [1] are used, where the silences in the manual segmentation from the official AMI release were stripped by using a forced alignment with the reference transcriptions and a preexisting speech recognition system [32]. These alignments were also used to generate the speech/nonspeech labels for training the VAD.…”

Section: Experimental Setup 41 Datamentioning

confidence: 99%

“…Since these scenarios often result in long audio streams, prior to applying ASR, it is necessary to obtain individual speech segments by finding "who spoke when" using speaker diarisation. A speaker diarisation pipeline often consists of at least three modules, namely voice activity detection (VAD), speaker embedding extraction, and clustering, which are normally implemented with separate models [1]. A separately trained ASR system can then be used to transcribe each segment found by speaker diarisation, and obtain speaker-attributed ASR output over long audio streams [2,3].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Tandem Multitask Training of Speaker Diarisation and Speech Recognition for Meeting Transcription

Zheng¹,

Zhang²,

Woodland³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Self-supervised-learning-based pre-trained models for speech data, such as Wav2Vec 2.0 (W2V2), have become the backbone of many speech tasks. In this paper, to achieve speaker diarisation and speech recognition using a single model, a tandem multitask training (TMT) method is proposed to fine-tune W2V2. For speaker diarisation, the tasks of voice activity detection (VAD) and speaker classification (SC) are required, and connectionist temporal classification (CTC) is used for ASR. The multitask framework implements VAD, SC, and ASR using an early layer, middle layer, and late layer of W2V2, which coincides with the order of segmenting the audio with VAD, clustering the segments based on speaker embeddings, and transcribing each segment with ASR. Experimental results on the augmented multi-party (AMI) dataset showed that using different W2V2 layers for VAD, SC, and ASR from the earlier to later layers for TMT not only saves computational cost, but also reduces diarisation error rates (DERs). Joint fine-tuning of VAD, SC, and ASR yielded 16%/17% relative reductions of DER with manual/automatic segmentation respectively, and consistent reductions in speaker attributed word error rate, compared to the baseline with separately fine-tuned models.

show abstract

Section: Sc For Speaker Embedding Extraction and Clusteringmentioning

confidence: 99%

Section: Experimental Setup 41 Datamentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Tandem Multitask Training of Speaker Diarisation and Speech Recognition for Meeting Transcription

Zheng¹,

Zhang²,

Woodland³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…M ULTI-PARTY interactions such as meetings and conversations are one of the most important scenarios for many speech and language applications [1]. Speaker change detection (SCD), the task of finding the time points that a new speaker starts to speak, is critical for such applications and has received increasing attention in recent years [2]- [5].…”

Section: Introductionmentioning

confidence: 99%

Sequence-Level Speaker Change Detection With Difference-Based Continuous Integrate-and-Fire

Fan

Dong²,

Cai³

et al. 2022

IEEE Signal Process. Lett.

View full text Add to dashboard Cite

Speaker change detection is an important task in multi-party interactions such as meetings and conversations. In this paper, we address the speaker change detection task from the perspective of sequence transduction. Specifically, we propose a novel encoder-decoder framework that directly converts the input feature sequence to the speaker identity sequence. The differencebased continuous integrate-and-fire mechanism is designed to support this framework. It detects speaker changes by integrating the speaker difference between the encoder outputs frame-byframe and transfers encoder outputs to segment-level speaker embeddings according to the detected speaker changes. The whole framework is supervised by the speaker identity sequence, a weaker label than the precise speaker change points. The experiments on the AMI and DIHARD-I corpora show that our sequence-level method consistently outperforms a strong framelevel baseline that uses the precise speaker change labels.

show abstract

“…Given that there is empirically no unique outstanding acoustic features or models suitable for various test scenarios, another thought for improving the performance of speaker recognition is to combine complementary acoustic features or models. Sun et al [17] proposed a c-vector method by combining multiple sets of complementary d-vectors derived from systems with different neural network components, including 2-dimensional self-attentive, gated additive, bilinear pooling structures, etc. Language recognition shares quite a similar research trend with speaker recognition.…”

mentioning

confidence: 99%

Guest editorial: Special issue on advances in deep learning based speech processing

2023

View full text Add to dashboard Cite

Combination of deep speaker embeddings for diarisation

Cited by 11 publications

References 91 publications

Tandem Multitask Training of Speaker Diarisation and Speech Recognition for Meeting Transcription

Tandem Multitask Training of Speaker Diarisation and Speech Recognition for Meeting Transcription

Sequence-Level Speaker Change Detection With Difference-Based Continuous Integrate-and-Fire

Guest editorial: Special issue on advances in deep learning based speech processing

Contact Info

Product

Resources

About