Predictability of Word Forms (Types) and Lemmas in Linguistic Corpora. A Case Study Based on the Analysis of the CUMBRE Corpus

Abstract

Various research centres and publishing companies all around the world have been developing corpus resources for many years, and there has been a growing awareness throughout the eighties of their importance to linguistic and lexicographic work. To give some idea of scale, the British National Corpus contains 100 million words, and its counterpart for Spanish—compiled by the Spanish Real Academia de la Lengua—will reach 100 million words at first and 200 million words in a second stage. However, little convincing research has been done in the direction of sample size—directly connected to a further topic: representativeness. We shall investigate here a related issue: Is it possible to predict the different word forms and lemmas of a given corpus? And if so, how? A positive answer to this question may contribute to decision making regarding some aspects of representativeness in given fields. We shall attempt further to find a reliable procedure to predict the total number of word forms (types) and lemmas in a specific corpus.

An Algorithm for Predicting the Relation between Lemmas and Corpus Size

“…billion tokens) is needed, we might put more emphasis on balance and coverage; (c) to induce the relationship between lemmas and tokens, types and tokens, and lemmas and types. The usefulness of these dependencies becomes evident in many ways, in particular whenever we want to contrast regularity patterns regarding specific linguistic areas (e.g., written versus spoken, Spanish versus Korean, newspapers versus fiction, see [13], [20]; and finally (d) to represent any monotone, convex up, increasing curve that cannot be represented by single functions. That is, whenever the error term is too big to be represented by a single function.…”

“…This suggests that even if we were to find a function that could fit some given data (corpus), there would still be no guarantee that the function would always hold. This is precisely the common major flaw of Young-Mi Jeong's and Sánchez and Cantos' researches within the field of computational linguistics [12], [13]. In Section III-3, we shall delineate experimentally this problem in more detail.…”

“…We are confident that Heaps, Young-Mi Jeong and Sánchez and Cantos failed by using regression techniques [11]- [13]. This might be attributed to their preference for Zipf's laws and on their belief that these statistical models would not just hold for their moderately large experimental data but also for any data (corpus size).…”

“…Heaps, Young-Mi Jeong, and Sánchez and Cantos experimented with small texts and/or relatively small corpora [11]- [13]. However, as already discussed, empirical corpus-based data depend heavily on the corpora from which this information has been extracted.…”

“…Notice that Fig. 4 (a) visually represents the limitations and problems of the predicting approaches used by Heaps, Young-Mi Jeong and Sánchez and Cantos [11]- [13]. A solution to this problem is to predict only for the interval [3300000, 50000000], leaving aside the interval [1,3300000].…”