A comparison of document, sentence, and term event spaces

Abstract

The trend in information retrieval systems is from document to sub-document retrieval, such as sentences in a summarization system and words or phrases in question-answering system. Despite this trend, systems continue to model language at a document level using the inverse document frequency (IDF). In this paper, we compare and contrast IDF with inverse sentence frequency (ISF) and inverse term frequency (ITF). A direct comparison reveals that all language models are highly correlated; however, the average ISF and ITF values are 5.5 and 10.4 higher than IDF. All language models appeared to follow a power law distribution with a slope coefficient of 1.6 for documents and 1.7 for sentences and terms. We conclude with an analysis of IDF stability with respect to random, journal, and section partitions of the 100,830 full-text scientific articles in our experimental corpus.

Text Clustering for Peer-to-Peer Networks with Probabilistic Guarantees

“…PCP2P relies on the fact that term frequencies follow a Zipf distribution. Although it is accepted that document collections follow Zipf distribution, different document collections exhibit different distribution skews [1]. For example, the RCV1 collection used in our experiments has a skew of 0.55, while values reported in the literature for other text collections are around 1.0 [1,17].…”

“…Although it is accepted that document collections follow Zipf distribution, different document collections exhibit different distribution skews [1]. For example, the RCV1 collection used in our experiments has a skew of 0.55, while values reported in the literature for other text collections are around 1.0 [1,17]. To evaluate the influence of the skew on PCP2P, we used SYNTH collections generated with different Zipf skew factors, between 0.5 and 1.1.…”

“…The equations follow directly from Chernoff inequality, similar to Theorem 1. To compute the expected term frequencies for tx[φi], i.e., the x'th most frequent term in φi, peers use the Zipf distribution (validated, for example, in [1] for text).…”

PCIR: Combining DHTs and peer clusters for efficient full-text P2P indexing

“…For this we assume, as is common in IR [22,23], that the dictionary size of each peer follows Heap's law [23]. We denote the length of a document, i.e., the number of words it consists of, as len(d), and the length of a document collection len(DC(p)) = d∈DC(p) len(d).…”

The role of statistical and semantic features in single-document extractive summarization

“…Blake [13] compares different language models and speculates on using ISF for the systems aiming at sentence extraction and inverse term frequency (ITF) for the systems identifying terms as their smallest compositional unit. Thus for the TS systems based on unigrams ITF could be a reliable method to select the sentences for the final summary.…”