Multi-speaker language modeling

Abstract

In conventional language modeling, the words from only one speaker at a time are represented, even for conversational tasks such as meetings and telephone calls. In a conversational or meeting setting, however, speakers can have significant influence on each other. To recover such un-modeled inter-speaker information, we introduce an approach for conversational language modeling that considers words from other speakers when predicting words from the current one. By augmenting a normal trigram context, our new multi-speaker language model (MSLM) improves on both Switchboard and ICSI Meeting Recorder corpora. Using an MSLM and a conditional mutual information based word clustering algorithm, we achieve a 8.9% perplexity reduction on Switchboard and a 12.2% reduction on the ICSI Meeting Recorder data.

Morphology-based language modeling for conversational Arabic speech recognition

“…Since their inception at the Johns-Hopkins University Summer Workshop in 2002, FLMs have been used successfully for various language modeling and speech processing tasks (Bilmes and Kirchhoff, 2003;Parandekar and Kirchhoff, 2003;Ji and Bilmes, 2004). For the present task we tested several FLM structures (different sets of conditioning factors and various backoff paths) manually to optimize the perplexity on the development set.…”

Graphical Model Representations of Word Lattices

“…In the long run, we wish to use GMTK for a variety of novel LVCSR tasks, including re-scoring using multi-speaker LMs [6]. Therefore, our lattices are unique in that they were strung together to represent one side of an entire conversation.…”

“…This type of model often represents a variety of explicit and intricate aspects of the speech signal (such as articulatory features [5], or various crossspeaker or multi-stream information [6]). Algorithmic efforts to enable these complex models to produce first-pass hypotheses is a worthy research goal, but we may simultaneously evaluate certain aspects of these models using a lattice to limit the state space, something that does not require the more complex inference procedures.…”

Session-level Language Modeling for Conversational Speech

“…This presumes that each utterance is independent of the others, and clearly violates what we know about how language and conversation works, as discussed in the next section. Consequently, there have been many proposals to inject information from a longer context into standard LM architectures, going back to Ngram models (Bellegarda, 2004), or to generalize N-grams LMs to operate across utterance boundaries and speakers (Ji and Bilmes, 2004). Based on the RNN framework, (Mikolov and Zweig, 2012) proposed augmenting network inputs with a more slowly varying context vector that would encode longer-range properties of the history, such as a latent semantic indexing vector.…”