Speaker Adaptation for Multichannel End-to-End Speech Recognition

Ochiai, Tsubasa; Watanabe, Shinji; Katagiri, Shigeru; Hori, Takaaki; Hershey, John R.

doi:10.1109/icassp.2018.8462161

Cited by 94 publications

(106 citation statements)

References 31 publications

Supporting

Mentioning

106

Contrasting

Order By: Relevance

“…We used a similar architecture as in [23], where the masking network and the neural beamformer are integrated into an attentionbased encoder-decoder neural network, and the whole model is jointly optimized solely via a speech recognition objective. The input of the model can consist of an arbitrary number of channels C, and its output is the text sequence for each speaker directly.…”

Section: End-to-end Multi-channel Multi-speaker Asrmentioning

confidence: 99%

MIMO-Speech: End-to-End Multi-Channel Multi-Speaker Speech Recognition

Chang

Zhang

et al. 2019

2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

Self Cite

103

View full text Add to dashboard Cite

Recently, the end-to-end approach has proven its efficacy in monaural multi-speaker speech recognition. However, high word error rates (WERs) still prevent these systems from being used in practical applications. On the other hand, the spatial information in multi-channel signals has proven helpful in far-field speech recognition tasks. In this work, we propose a novel neural sequence-tosequence (seq2seq) architecture, MIMO-Speech, which extends the original seq2seq to deal with multi-channel input and multi-channel output so that it can fully model multi-channel multi-speaker speech separation and recognition. MIMO-Speech is a fully neural end-toend framework, which is optimized only via an ASR criterion. It is comprised of: 1) a monaural masking network, 2) a multi-source neural beamformer, and 3) a multi-output speech recognition model. With this processing, the input overlapped speech is directly mapped to text sequences. We further adopted a curriculum learning strategy, making the best use of the training set to improve the performance. The experiments on the spatialized wsj1-2mix corpus show that our model can achieve more than 60% WER reduction compared to the single-channel system with high quality enhanced signals (SI-SDR = 23.1 dB) obtained by the above separation function.

show abstract

Section: End-to-end Multi-channel Multi-speaker Asrmentioning

confidence: 99%

MIMO-Speech: End-to-End Multi-Channel Multi-Speaker Speech Recognition

Chang

Zhang

et al. 2019

2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

Self Cite

103

View full text Add to dashboard Cite

show abstract

“…There have been few efforts on the adaptation of the E2E systems. [22] proposed a multi-path adaptation scheme for end-to-end multichannel ASR. In [23], the authors addressed the data sparsity issue by formulating Kullback-Leibler divergence (KLD) regularization and multi-task learning approaches for speaker adaptation of CTC models.…”

Section: Related Workmentioning

confidence: 99%

Speaker-Aware Speech-Transformer

Fan

Zhou

et al. 2019

2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

View full text Add to dashboard Cite

Recently, end-to-end (E2E) models become a competitive alternative to the conventional hybrid automatic speech recognition (ASR) systems. However, they still suffer from speaker mismatch in training and testing condition. In this paper, we use Speech-Transformer (ST) as the study platform to investigate speaker aware training of E2E models. We propose a model called Speaker-Aware Speech-Transformer (SAST), which is a standard ST equipped with a speaker attention module (SAM). The SAM has a static speaker knowledge block (SKB) that is made of i-vectors. At each time step, the encoder output attends to the i-vectors in the block, and generates a weighted combined speaker embedding vector, which helps the model to normalize the speaker variations. The SAST model trained in this way becomes independent of specific training speakers and thus generalizes better to unseen testing speakers. We investigate different factors of SAM. Experimental results on the AISHELL-1 task show that SAST achieves a relative 6.5% CER reduction (CERR) over the speaker-independent (SI) baseline. Moreover, we demonstrate that SAST still works quite well even if the i-vectors in SKB all come from a different data source other than the acoustic training set.

show abstract

“…Although it achieved excellent performance as the front-end of ASR [8][9][10], it has a few drawbacks. First, the DNN was often trained to minimize the estimation error of T-F masks instead of maximizing the quality of the estimated signal directly [11,12]. In addition, its performance is limited under noisy and reverberant environment because it does not consider non-stationary characteristics of speech signal [13].…”

Section: Introductionmentioning

confidence: 99%

Consistency-Aware Multi-Channel Speech Enhancement Using Deep Neural Networks

Masuyama

Togami

Komatsu

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

This paper proposes a deep neural network (DNN)-based multichannel speech enhancement system in which a DNN is trained to maximize the quality of the enhanced time-domain signal. DNNbased multi-channel speech enhancement is often conducted in the time-frequency (T-F) domain because spatial filtering can be efficiently implemented in the T-F domain. In such a case, ordinary objective functions are computed on the estimated T-F mask or spectrogram. However, the estimated spectrogram is often inconsistent, and its amplitude and phase may change when the spectrogram is converted back to the time-domain. That is, the objective function does not evaluate the enhanced time-domain signal properly. To address this problem, we propose to use an objective function defined on the reconstructed time-domain signal. Specifically, speech enhancement is conducted by multi-channel Wiener filtering in the T-F domain, and its result is converted back to the time-domain. We propose two objective functions computed on the reconstructed signal where the first one is defined in the time-domain, and the other one is defined in the T-F domain. Our experiment demonstrates the effectiveness of the proposed system comparing to T-F masking and mask-based beamforming.

show abstract

Speaker Adaptation for Multichannel End-to-End Speech Recognition

Cited by 94 publications

References 31 publications

MIMO-Speech: End-to-End Multi-Channel Multi-Speaker Speech Recognition

MIMO-Speech: End-to-End Multi-Channel Multi-Speaker Speech Recognition

Speaker-Aware Speech-Transformer

Consistency-Aware Multi-Channel Speech Enhancement Using Deep Neural Networks

Contact Info

Product

Resources

About