An Empirical Model for n-gram Frequency Distribution in Large Corpora

Abstract

Statistical multiword extraction methods can benefit from the knowledge on the n-gram (n ≥ 1) frequency distribution in natural language corpora, for indexing and time/space optimization purposes. The appearance of increasingly large corpora raises new challenges on the investigation of the large scale behavior of the n-gram frequency distributions, not typically emerging on small scale corpora. We propose an empirical model, based on the assumption of finite n-gram language vocabularies, to estimate the number of distinct n-grams in large corpora, as well as the sizes of the equal-frequency n-gram groups, which occur in the lower frequencies starting from 1. The model was validated for n-grams with 1 ≤ n ≤ 6, by a wide range of real corpora in English and French, from 60 million up to 8 billion words. These are full nontruncated corpora data, that is, their associated frequency data include the entire range of observed n-gram frequencies, from 1 up to the maximum. The model predicts the monotonic growth of the numbers of distinct n-grams until reaching asymptotic plateaux when the corpus size grows to infinity. It also predicts the non-monotonicity of the sizes of the equal-frequency n-gram groups as a function of the corpus size.

Categorisation of Computer Science Research Papers using Supervised Machine Learning Techniques

“…Bag-of-words approaches typically use 1-grams as features to represent text, while bag-ofterms approaches use a combination of n-grams. According to Silva and Cunha, n-grams do not produce major improvements in the classification performance, but the actual impact on often dependent on the application [14]. http://journals.uob.edu.bh Dimensionality reduction can be considered as a compression of the data.…”