An Efficient Approach for Inverted Index Pruning Based on Document Relevance

Abstract

Information Retrieval deals with retrieving documents from a large collection that matches the information need of a user. Efficient retrieval is based on the proper storage of the inverted index. There have been many techniques for reducing the size of the inverted index. Static index pruning is one such technique, which is used to reduce the index size. This paper investigates a static index pruning approach which is useful to reduce the index size. The proposed approach prunes the entire document from the index based on its importance and relevance of top-k results. The elimination takes place on the basis of the score of the individual document. Experiments have been conducted on the FIRE text collection. Based on the results, it was found that for specific collections, the proposed model gives better precision values for the retrieval of top 30 and above documents.

A Scalable Document-Based Architecture for Text Analysis

“…In the various methods for managing inverted indexes, great emphasis is put on storage space reduction. For instance, a pruning algorithm based on term frequency-inverse document frequency (TF*IDF) can be used to minimize index size [19]. Yet, updating an inverted index is also a problem, because it is dependent on documents.…”