Split size‐rank models for the distribution of index terms

Abstract

Since the introduction of the Zipf distribution, many functions have been suggested for the frequency of words in text. Some of these models have also been a p piled to the distribution of Index terms in a set of documents. The models are of two forms: rank-frequency and frequency-size. The former serve well to describe the distribution of high-frequency terms; the latter the distribution of low-frequency terms. In this article, a split model is proposed, which uses both a rank function for the high frequency terms and a size function for the low frequency terms, with the point of transition being determined either empirically or by rule. This model is fitted to the marginal empirical term distributions for four document datasets. Distributions to describe index term exhaustivity and term co-occurrence are also considered briefly.

Yes, the GIGP really does work—and is workable!

“…Index terms featuring in the "MEDLARS" database (Nelson & Tague, 1985 processes is not an appropriate model for these. In others, though, we found most acceptable fits.…”

“…(j) "MEDLARS" (Nelson & Tague, 1985) When first presented, the long-tailed nature of these data obliged Nelson and Tague to seek different sorts of distribution to model the low-to-medium frequencies and the high frequencies. Sichel (1992) achieved an acceptable fit for the GIGP with (Y -0 assumed a priori, based on sample mean and proportion of singletons, and using the originally reported groupings.…”

The influence of indexing practices and weighting algorithms on document spaces

“…Figure 1 shows the exhaustivity distributions used in the study representing low (M = 7 terms per document), observed (M ϭ 11 terms per document), and high (M ϭ 15 terms per document) levels of exhaustivity. The hypothetical low and high exhaustivity distributions were based on negative binomial models, which have been shown to be representative of exhaustivity in actual environments (Bird, 1974;Nelson & Tague, 1985). Figure 2 shows the distribution of descriptor occurrences across the document set; that is, the number of documents that contain a given descriptor.…”

Retrieval testing with hypergeometric document models

“…The modeling of databases is not new but a realistic simulation of the relevance relation has not previously been known. Griffiths (1978); Tague, Nelson, and Wu (1981) ;Tague, McClellan, & Nelson (1984); Nelson (1981, 1982); and Nelson and Tague (1985) have developed models of bibliographic databases. They have attempted to model the statistics of term occurrence within documents and globally throughout a whole collection.…”