ASR-aware end-to-end neural diarization

Abstract

We present a Conformer-based end-to-end neural diarization (EEND) model that uses both acoustic input and features derived from an automatic speech recognition (ASR) model. Two categories of features are explored: features derived directly from ASR output (phones, position-in-word and word boundaries) and features derived from a lexical speaker change detection model, trained by fine-tuning a pretrained BERT model on the ASR output. Three modifications to the Conformer-based EEND architecture are proposed to incorporate the features. First, ASR features are concatenated with acoustic features. Second, we propose a new attention mechanism called contextualized self-attention that utilizes ASR features to build robust speaker representations. Finally, multi-task learning is used to train the model to minimize classification loss for the ASR features along with diarization loss. Experiments on the two-speaker English conversations of Switchboard+SRE data sets show that multi-task learning with position-in-word information is the most effective way of utilizing ASR features, reducing the diarization error rate (DER) by 20\% relative to the baseline.

Bertraffic: Bert-Based Joint Speaker Role and Speaker Change Detection for Air Traffic Control Communications

“…Later, the same authors upgraded the system by replacing the bidirectional long short-term memory (BLSTM) layers by self-attention modules [19]. Subsequent work has targeted EEND for unknown number of speakers [20], SD for long conversations [21], streaming EEND [22], SD constrained by turn detection (i.e., SCD) in [23], or even leveraging EEND for ASR [24].…”

“…In [32] a text-based SRD for multiparty dialogues is proposed, but limited to SRD. Finally, text-based diarization has been proposed in the past by [22,24]. However, these previous works do not take into account the text structure, grammar, and syntax.…”

DOVER-Lap: A Method for Combining Overlap-Aware Diarization Outputs

“…Combing the auxiliary encoder representation with representations from the ASR prediction network allows the speaker branch to leverage lexical content for predicting speaker labels. Such a use of lexical information has been shown to be beneficial for speaker diarization using clusteringbased [32,33] or end-to-end neural approaches [34].…”

Tandem Multitask Training of Speaker Diarisation and Speech Recognition for Meeting Transcription

“…A separately trained ASR system can then be used to transcribe each segment found by speaker diarisation, and obtain speaker-attributed ASR output over long audio streams [2,3]. Recently, end-to-end methods have been proposed for jointly modelling some modules in a speaker diarisation pipeline with an ASR system [4][5][6][7][8][9][10][11][12].…”