A document retrieval system based on nearest neighbour searching

Abstract

Document filing and retrieval systems can be designed using advanced techniques resulting from recent research in information retneval. In this paper, a document retneval system is presented, based upon the vector processing model. The system employs an automatic indexing procedure with a weighting scheme to reflect term importance. Documents are stored using an in verted file organization. Natural language quenes are sup ported with a retrieval strategy based on best match techniques and relevance feedback. The emphasis is on nearest neighbour searching to locate documents closest to a given query. That means, after having defined a sirrularitv function, the identification of those docu ments in the collection which exhibit a higher degree of re semblance to the query. The problem is introduced with reference to a straightfor ward search procedure that returns the nearest neighbour set manipulating the inverted file entnes. Then. an improved al gorithm is presented which optimizes both the number of documents to be evaluated and the number of inverted lists to be inspected.

Memory efficient ranking

“…In an inverted file document database, each distinct word in the database is held in a vocabulary [4,8,11,19]. The vocabulary might be an array, or a search structure such as a B-tree that can be efficiently updated.…”

“…This is particularly important for boolean queries-that is, queries for which the answers are exactly those documents that contain all of the terms specified in the query-as it allows the space and time needed to find the intersection of the inverted file entries to be minimised. For ranked queries, where the list operation is effectively an 'or' rather than an 'and', knowledge of the number of items in the inverted file entries is also useful, and allows several time-saving heuristics to be employed [4,11].…”

“…Another method for determining the ranking is to directly use the inverted file structure and document lengths [4,8,11], as we choose to do. In this method, the index entry for each word in the query is retrieved in turn.…”

“…In some cases, it is not necessary to access the index entries for words of low weight because they are unlikely to affect the ranking, and so a possible heuristic is to access the entries in decreasing order of weight [18], as is done to minimise processing time for boolean queries. Buckley and Lewit [4] and Lucarella [11] have further explored this possibility.…”

“…One method that has been suggested for avoiding this bottleneck is to store, in the inverted file entries, not the raw f d,t values described above, but instead [4,11]. Then, as the partial similarities are built up in the accumulators, they will be 'self-normalising', and no explicit access to the document lengths will be required.…”