Whole-sentence exponential language models: a vehicle for linguistic-statistical integration

Abstract

We introduce an exponential language model which models a whole sentence or utterance as a single unit. By avoiding the chain rule, the model treats each sentence as a "bag of features", where features are arbitrary computable properties of the sentence. The new model is computationally more efficient, and more naturally suited to modeling global sentential phenomena, than the conditional exponential (e.g. maximum entropy) models proposed to date. Using the model is straightforward. Training the model requires sampling from an exponential distribution. We describe the challenge of applying Monte Carlo Markov Chain and other sampling techniques to natural language, and discuss smoothing and step-size selection. We then present a novel procedure for feature selection, which exploits discrepancies between the existing model and the training corpus. We demonstrate our ideas by constructing and analysing competitive models in the Switchboard and Broadcast News domains, incorporating lexical and syntactic information.

Comparing human and automatic speech recognition in a perceptual restoration experiment

“…Future experiments are needed to evaluate system performance on isolated words, and the performance on isolated words and complete sentences must be compared in order to determine whether the additional context information available in complete sentences improves system performance. Since previous studies indicate that semantic context is important to human listeners in the perceptual restoration task, future experimenters may wish to evaluate top-down completion approaches with a whole-sentence language model (Rosenfeld et al, 2001).…”

Semantic language models with deep neural networks

“…Another type of maximum entropy LMs is the whole sentence model given in [111]. This model predicts the probability of the whole sentence by using feature functions over the entire sentence.…”

N-Best Re-scoring Approaches for Mandarin Speech Recognition

“…Different form generative models requiring the joint probability of the observations and the labels, it calculates the conditional probability of labels over the observations directly. Rosenfeld proposed a maximum entropy based language model that employed shallow syntactic features [8]. Magdin used Maximum Mutual Information Estimation (MMIE) method to optimize an objective function that involved a metric between correct transcriptions and their competing hypotheses, which are encoded as word graphs generated by Viterbi decoding process [18].…”

“…N-gram language model can be improved by directly integrated with higher level models, such as dependency relations and probabilistic top-down parsing [6,7]. Discriminative methods such as maximum entropy models can also be used to utilize global features from word sequences and syntactic structures [8]. Discriminative re-scoring methods…”