Combining Frame-Synchronous and Label-Synchronous Systems for Speech Recognition

Abstract

Commonly used automatic speech recognition (ASR) systems can be classified into frame-synchronous and labelsynchronous categories, based on whether the speech is decoded on a per-frame or per-label basis. Frame-synchronous systems, such as traditional hidden Markov model systems, can easily incorporate existing knowledge and can support streaming ASR applications. Label-synchronous systems, based on attentionbased encoder-decoder models, can jointly learn the acoustic and language information with a single model, which can be regarded as audio-grounded language models. In this paper, we propose rescoring the 𝑁-best hypotheses or lattices produced by a firstpass frame-synchronous system with a label-synchronous system in a second-pass. By exploiting the complementary modelling of the different approaches, the combined two-pass systems achieve competitive performance without using any extra speech or text data on two standard ASR tasks. For the 80-hour AMI IHM dataset, the combined system has a 13.7% word error rate (WER) on the evaluation set, which is up to a 29% relative WER reduction over the individual systems. For the 300-hour Switchboard dataset, the WERs of the combined system are 5.7% and 12.1% on Switchboard and CallHome subsets of Hub5'00, and 13.2% and 7.6% on Switchboard Cellular and Fisher subsets of RT03, up to a 33% relative reduction in WER over the individual systems.

Have best of both worlds: two-pass hybrid and E2E cascading framework for speech recognition

“…Given that both hybrid and E2E modeling have their own advantages, a natural question to ask is whether we can combine them to have the best of both worlds. Several publications explored this direction [17,18,19]. In these works, hybrid and E2E models are trained independently, and hypothesis level combination between the two systems is carried out by either rescoring the N-best and lattice from hybrid system with an E2E model [17,18], or Minimum Bayes' Risk (MBR) combination of the N-best lists of the two systems [19].…”