Multi-Stream End-to-End Speech Recognition

Abstract

Attention-based methods and Connectionist Temporal Classification (CTC) network have been promising research directions for end-to-end (E2E) Automatic Speech Recognition (ASR). The joint CTC/Attention model has achieved great success by utilizing both architectures during multi-task training and joint decoding. In this work, we present a multi-stream framework based on joint CTC/Attention E2E ASR with parallel streams represented by separate encoders aiming to capture diverse information. On top of the regular attention networks, the Hierarchical Attention Network (HAN) is introduced to steer the decoder toward the most informative encoders. A separate CTC network is assigned to each stream to force monotonic alignments. Two representative framework have been proposed and discussed, which are Multi-Encoder Multi-Resolution (MEM-Res) framework and Multi-Encoder Multi-Array (MEM-Array) framework, respectively. In MEM-Res framework, two heterogeneous encoders with different architectures, temporal resolutions and separate CTC networks work in parallel to extract complimentary information from same acoustics. Experiments are conducted on Wall Street Journal (WSJ) and CHiME-4, resulting in relative Word Error Rate (WER) reduction of 18.0 − 32.1% and the best WER of 3.6% in the WSJ eval92 test set. The MEM-Array framework aims at improving the farfield ASR robustness using multiple microphone arrays which are activated by separate encoders. Compared with the best singlearray results, the proposed framework has achieved relative WER reduction of 3.7% and 9.7% in AMI and DIRHA multiarray corpora, respectively, which also outperforms conventional fusion strategies.

A Practical Two-Stage Training Strategy for Multi-Stream End-to-End Speech Recognition

“…An end-to-end ASR model addressing the general multi-stream setting was introduced in [23]. As one representative framework, MEM-Array concentrates on cases of far-field microphone arrays to handle different dynamics of streams.…”

“…In [23], we proposed a novel multi-stream model based on a joint CTC/Attention E2E scheme, where each stream is characterized by a separate encoder and CTC network. A Hierarchical Attention Network (HAN) [24,25] acts as a fusion component to dynamically assign higher weights for streams carrying more discriminative information for prediction.…”

A Pronunciation Prior Assisted Vowel Reduction Detection Framework with Multi-Stream Attention Method

“…In other words, it is expected to use auxiliary information to determine possible vowel reduction positions, and at the same time, to detect whether there is vowel reduction in acoustic information. There are some methods for fusing multi-stream input, such as the direct merging of highdimensional feature vectors and the hierarchical attention dynamic fusion method used by [20,21]. However, these methods treat different inputs as independent streams.…”

Scaling Sparsemax Based Channel Selection for Speech Recognition with ad-hoc Microphone Arrays

“…The second kind aims to conduct channel selection and channel fusion for optimizing the ASR performance directly [8,9,12,13]. Early methods [8,9] chose the channels with the highest likelihood of the output after the decoding of ASR.…”