Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chain

Abstract

Non-autoregressive (NAR) models have achieved a large inference computation reduction and comparable results with autoregressive (AR) models on various sequence to sequence tasks. However, there has been limited research aiming to explore the NAR approaches on sequence to multi-sequence problems, like multi-speaker automatic speech recognition (ASR). In this study, we extend our proposed conditional chain model to NAR multi-speaker ASR. Specifically, the output of each speaker is inferred one-by-one using both the input mixture speech and previously-estimated conditional speaker features. In each step, a NAR connectionist temporal classification (CTC) encoder is used to perform parallel computation. With this design, the total inference steps will be restricted to the number of mixed speakers. Besides, we also adopt the Conformer and incorporate an intermediate CTC loss to improve the performance. Experiments on WSJ0-Mix and LibriMix corpora show that our model outperforms other NAR models with only a slight increase of latency, achieving WERs of 22.3% and 24.9%, respectively. Moreover, by including the data of variable numbers of speakers, our model can even better than the PIT-Conformer AR model with only 1/7 latency, obtaining WERs of 19.9% and 34.3% on WSJ0-2mix and WSJ0-3mix sets. All of our codes are publicly available at https://github.com/pengchengguo/espnet/tree/conditionalmultispk.

Conformer-Based Target-Speaker Automatic Speech Recognition For Single-Channel Audio

“…Table 1 compares performance of the proposed CONF-TSASR model with contemporary results on the WSJ0-mixextr datasets. For baselines, we include a conventional ASR model (Conformer-CTC), SpeakerBeam [10], Exformer [23] and Conditional-Conformer-CTC [6]. The first model was trained on single-speaker, the second and third were trained on two speakers and the last model was trained on up to three speakers.…”

“…As the separation step of BSS is not optimized for ASR, this can be sub-optimal. Multi-speaker ASR approaches [3,4,5,6] and their speaker-attributed variants (SA-ASR) [7,8] generate transcripts as output and are optimized end-to-end for ASR. A characteristic of BSS models and their analogous multi-speaker ASR models is their multiple output branches, one per source.…”

End-to-End Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysis

“…The main challenges faced in these scenarios include overlapping speech, reverberation caused by distant microphones, and background noise. End-to-end multi-speaker ASR and diarization systems for single-channel [1][2][3][4] and multichannel recordings [5][6][7][8] have recently emerged, demonstrating promising results on meeting transcription tasks.…”

“…The earlier approaches for end-to-end multi-speaker ASR and diarization for single-channel recordings lacked information sharing between the ASR and the speaker verification modules [2] and/or required a varying number of speaker encoder (or attention) modules based on the number of speakers [1,3]. The authors of [4] proposed an end-toend single-channel speaker-number invariant Transformerbased speaker-attributed ASR (SA-ASR) system based on serialized output training (SOT).…”

The NPU-ASLP System for The ISCSLP 2022 Magichub Code-Swiching ASR Challenge

“…By using a conditional chain, a mixed audio input with multiple speakers can be sequentially separated into individual outputs, where each output corresponds to a different speaker, with the previous output sequence serving as the conditional input. This approach can also be extended to multitalker ASR tasks to produce the transcription of different speakers directly [28,29]. The limitation of the conditional chain approach is computationally expensive, as it requires multiple iterations to separate each speaker signal from the mixed audio signal.…”