Some clumping experiments for associative document retrieval

Abstract

The paper describes a computational procedure for defining associations between words used to index a collection of documents. Using association structures generated by the procedure in a small document col-This paper describes experiments with a statistical technique that can be iised to compute word associations 119eful for document retrieval purposes in a collection where documents are described by index words occurring in the tcst or in abstracts of the documcnts.Although the experiments have been conducted with a small collection of documents, they have suggested methods that will permit the association technique to be applied in collections of an interesting size. This is important, since a number of alternative methods that have been proposed for computing word associations appear to present intractable computation problems that \vill deter large scale application. Why Associative Retrieval?There :ire three principal reasons why associative 1. The efficiency of retrieval in a coordinate indexed system rapidly diminishes as the document collection grows in size, I n an associative retrieval system, on the other hand, retrieval efficiency as measured by recall and relevance ratios will not necessarily decrease with the size of the document collection, and might, in fact, improve ( 1 ) . 2. If the structure of word associations in a given collection is adequate, the user of the collection can define a retrieval request in his own terms without reference to an elaborate dictionary or thesaurus of index terms. Thus, the need for interposing an interpreter between the user and the collection e m be dispensed with, and the ex-retrieval techniques are worth investigating:lection, a retrieval model is described and the results of trial retrievals are evaluated. The techniques described appear to be applicable in practice for retrieval in large collections.ponentially increasing difficulty of compiling an external thesaurus is avoided. 3. The techniques can be applied to word data that has been automatically processed from document tests, or identified in texts by individuals inesperienced in complex conventional indexing methods, thus permitting rapid indexing and acquisition of materials in a collection. Clumping Techniques and Word AssociationsThe method used to define word associations is based' on the theory of clumps (2,3,4,5), a technique developed largely by R. M. Needham of the Cambridge Language Research Unit. The notion of a clump is intuitively simple: a clump is a group of objects from some universe of objects such that the members of the group (tech. nically, the nibset) are more closely related to each other than to the rest of the universe to which they belong.I n order to find clumps of objects, two decisions must first be made: 1. We have to specify how associations between objects are to be measured. 2.We have to specify the criteria that define a clump.

Automatic Subject Recognition in Scientific Papers: An Empirical Study

“…6.27 The results in the other four tests were 100 percent recognition and five percent overassigning [21], 94 percent recognition and 40 percent overassigning [7t, 80 percent recognition and 10 percent overassigning [7], 80 percent recognition and 13 percent overassigning [27] and 60 percent recognition and 10 percent overassigning [31].…”

“…Some of these methods have been tested empirically [4,5,7,12,21,24,26,27,30,31]. The basis of each test has been comparison of subject specification-document pairs formed by knowledgeable humans and pairs formed by computer.…”

“…(1) The literature was in biomedicine, rather than in physics, the computer sciences or psychology. 1 (2) About 60 percent of the papers were longer than 1000 words (the average length 2000), rather than letters to the editor [27,26] or abstracts [4,5,7,12,21,24,30,31]. (3) Two subject specifications were studied intensively--for one specification, 300 documents humanly paired with it were examined, while for the other specification 55 papers paired with it were studied.…”

GPS and the Joint Force Commander: Critical Asset, Critical Vulnerability

“…Reports that low power, inexpensive jammers can defeat receivers have elicited great concern over the reliance on GPS for missile guidance. 22 Fears were fueled in recent years by the emergence of advanced commercially available jammers. During a 1997 Russian air show, for example, a $4,000 jamming transmitter was on display and its manufacturers claimed the device foils the ability of GPS receivers to provide correct geographic coordinates.…”

An Analysis of Some Graph Theoretical Cluster Techniques

“…Some measure of the relatedness between terms used to index the documents of the data set must be established in order to perform cluster analysis. Several different similarity measures have been proposed [7,12,18,21,41,43]. Since in-depth comparisons and evaluations concerning various similarity measures have been conducted before by other authors [18,21,36], only one similarity measure was studied.…”