Low Rank Language Models for Small Training Sets

Abstract

Abstract-Several language model smoothing techniques are available that are effective for a variety of tasks; however, training with small data sets is still difficult. This letter introduces the low rank language model, which uses a low rank tensor representation of joint probability distributions for parameter-tying and optimizes likelihood under a rank constraint. It obtains lower perplexity than standard smoothing techniques when the training set is small and also leads to perplexity reduction when used in domain adaptation via interpolation with a general, out-of-domain model.Index Terms-Language model, low rank tensor.

A sparse plus low rank maximum entropy language model

“…Thus, we anticipate that better feature sets will lead to even larger performance gains over the baseline. Motivated by previous results [1,2], we also expect that interpolating the SLR-LM with a standard smoothed n-gram model would yield further improvements.…”

“…For example, the cooccurrence statistics for a word α that has been observed 50 times may be sufficiently similar to a set of words β that have been observed hundreds or thousands of times for α's weights to be pushed into β's subspace of weights; in effect, this "fills in" missing entries from α's weight rows and columns. 1 The idea of learning and exploiting similarities between objects (e.g. words and histories) is a common theme in the literature on learning shared representations [5] and is used by language models with continuous representations of words [6,7].…”

“…While n-gram language modeling has been used with much success in applications where large training sets are available, there remains a need to better model small, targeted domains where limited data is available. We recently introduced a tensor-based "low rank language model" (LRLM) that outperforms baseline models when training data is limited [1]. A disadvantage of the LRLM is that the non-convex objective complicates training.…”

“…1), which had at least 350K word tokens. For each topic, we split the data (at the granularity of conversation) into training, development and test sets, at a 60/20/20 ratio.…”

Language Modeling with Power Low Rank Ensembles

“…There are a few low rank approaches (Saul and Pereira, 1997;Bellegarda, 2000;Hutchinson et al, 2011), but they are only effective in restricted set-tings (e.g. small training sets, or corpora divided into documents) and do not generally perform comparably to state-of-the-art models.…”