Learning to Count Words in Fluent Speech enables Online Speech Recognition

Abstract

Sequence to Sequence models, in particular the Transformer, achieve state of the art results in Automatic Speech Recognition. Practical usage is however limited to cases where full utterance latency is acceptable. In this work we introduce Taris, a Transformer-based online speech recognition system aided by an auxiliary task of incremental word counting. We use the cumulative word sum to dynamically segment speech and enable its eager decoding into words. Experiments performed on the LRS2 and LibriSpeech datasets, of unconstrained and read speech respectively, show that the online system performs on a par with the offline one, while having a dynamic algorithmic delay of 5 segments. Furthermore, we show that the estimated segment length distribution resembles the word length distribution obtained with forced alignment, although our system does not require an exact segment-to-word equivalence. Taris introduces a negligible overhead compared to a standard Transformer, while the local relationship modelling between inputs and outputs grants invariance to sequence length by design.

AV Taris: Online Audio-Visual Speech Recognition

“…We are interested in studying if the word count in fluent speech can be estimated with a higher accuracy from audio-visual cues than from the audio modality alone. In Sterpu et al (2021) we saw that the encoding look-ahead length does not have a major influence on either the word counting error or the character error rate. Therefore, in this experiment we limit our analysis to counting words from audio-visual representations with the offline models having infinite context available.…”

“…We begin this section by first reviewing the underlying approach Taris presented in Sterpu et al (2021).…”

“…For the reasons explained in Sterpu et al (2021), we do not use the common End-of-Sentence (EOS) token to pad our target sequences when training our online models.…”

“…The decoder in our previous experiment had access to the entire encoder memory. For our online models in this section we opt for an encoder look-ahead e LA of 11 frames and infinite look-back e LB = ∞, since we showed in Sterpu et al (2021) that there are diminishing gains beyond this threshold. In that work we have also demonstrated that Taris can leverage the gating signal α to limit the dynamic range of decoder-encoder attention and still match the error rate of the offline Transformer.…”

“…In our prior work Sterpu et al (2021), we investigated the research question of whether neural networks can learn to segment a spoken utterance by learning to count the words therein. Different from the way humans acquire a language, we suspected that there is a strong relationship between learning the ability to count spoken words, and the ability to segment words in fluent speech.…”