Performance of inverted indices in shared-nothing distributed text document information retrieval systems

Abstract

The performance of distributed text document retrieval systems is strongly in uenced b y t h e o r ganization of the inverted index. This paper compares the performance impact on query processing of various physical organizations for inverted lists. We present a new probabilistic model of the database and queries. Simulation experiments determine which variables most strongly inuence r esponse time and throughput. This lea d s t o a set of design trade-o s over a range of hardware c on gurations and new parallel query processing strategies.

Performance Analysis of Distributed Architectures to Index One Terabyte of Text

“…The regression analysis performed confirms that the quadratic model fits better the real distribution (R = 0.99770) versus the linear model representing the Zipf's law (R = 0.98122). The quadratic model is similar to Zipf's, although in previous works [15], it has proved to match the actual distribution better. Given the quadratic fit curve, the form of the probability distribution Z 1 (w) is obtained from the quadratic model, divided by a normalisation constant [15].…”

“…The previous work for distributing the inverted index over a collection of servers is focused on the local and global inverted files strategies [13], [15], showing that the local inverted file is a more balanced strategy and a good query throughput could be achieved in most cases.…”

“…If we are considering the whole collection Documents = D, but in a distributed environment, Documents corresponds to the number of documents covered by each of the distributed indices. So, the number of documents of an inverted list for term t i will be [15]:…”

“…The other option is to partition based on the index terms so that each query server stores inverted lists corresponding to only a subset of the index terms in the collection (called global inverted files in [13]). The study in [15] indicates that the local inverted file organization uses system resources effectively, provides good query throughput and is more resilient to failures.…”

ImprovingWeb Searches with Distributed Buckets Structures

“…The work presented in [26] compares the impact of performance for queries processing, using two different organizations for the invested lists. It proposes two basic options to classify the indexes: disk index and system index.…”