The Use of the Maximum Likelihood Criterion in Language Modelling

Ney, Hermann

doi:10.1007/978-3-642-60087-6_25

Search citation statements

Order By: Relevance

Paper Sections

Select...

N-gram Zipf Curves1

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2003

2009

Publication Types

Select...

Article1

Other1

Relationship

Self Cite0

Independent2

Authors

Journals

Cited by 2 publications

(1 citation statement)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These probabilities have been used with information theory in modelling language with much success over the last 20 years (Jelinek and Mercer 1985;O'Boyle et al 1994;Ney 1999).…”

Section: N-gram Zipf Curvesmentioning

confidence: 99%

Extending Zipf’s law to n-grams for large corpora

Ha¹,

Hanna

et al. 2009

Artif Intell Rev

View full text Add to dashboard Cite

Experiments show that for a large corpus, Zipf's law does not hold for all ranks of words: the frequencies fall below those predicted by Zipf's law for ranks greater than about 5,000 word types in the English language and about 30,000 word types in the inflected languages Irish and Latin. It also does not hold for syllables or words in the syllable-based languages, Chinese or Vietnamese. However, when single words are combined together with word n-grams in one list and put in rank order, the frequency of tokens in the combined list extends Zipf's law with a slope close to −1 on a log-log plot in all five languages. Further experiments have demonstrated the validity of this extension of Zipf's law to n-grams of letters, phonemes or binary bits in English. It is shown theoretically that probability theory alone can predict this behavior in randomly created n-grams of binary bits.

show abstract

“…These probabilities have been used with information theory in modelling language with much success over the last 20 years (Jelinek and Mercer 1985;O'Boyle et al 1994;Ney 1999).…”

Section: N-gram Zipf Curvesmentioning

confidence: 99%