Seq-2-Seq based Refinement of ASR Output for Spoken Name Capture

Abstract

This paper reimagines some aspects of speech processing using speech encoders, specifically about extracting entities directly from speech, with no intermediate textual representation. In human-computer conversations, extracting entities such as names, postal addresses and email addresses from speech is a challenging task. In this paper, we study the impact of fine-tuning pre-trained speech encoders on extracting spoken entities in human-readable form directly from speech without the need for text transcription. We illustrate that such a direct approach optimizes the encoder to transcribe only the entity relevant portions of speech, ignoring the superfluous portions such as carrier phrases and spellings of entities. In the context of dialogs from an enterprise virtual agent, we demonstrate that the 1-step approach outperforms the typical 2-step cascade of first generating lexical transcriptions followed by text-based entity extraction for identifying spoken entities.

E2E Spoken Entity Extraction for Virtual Agents

“…This model achieves a word accuracy of 93.1% on a 28k utterance test set consisting of user utterances that are in response to "How may I help you?" opening prompt from various enterprise virtual agent applications (Singla et al, 2022).…”

“…In this 2-step approach, we first transcribe the speech provided by humans into text using same pretrained E2E ASR checkpoint used by (Singla et al, 2022). We then extract entities from the transcribed text by learning to translate using (transcription, entity) pairs.…”

“…Until recently, where (Singla et al, 2022) adapt a standard Seq-2-Seq architecture to extract person names from automatically transcribed text. However, this 2-step approach means two sys-tems to maintain and adapt, but also, possibly loss of acoustic-prosodic information.…”