Query-Driven Indexing for Scalable Peer-to-Peer Text Retrieval

Abstract

We present a query-driven algorithm for the distributed indexing of large document collections within structured P2P networks. To cope with bandwidth consumption that has been identified as the major problem for the standard P2P approach with single term indexing, we leverage a distributed index that stores up to top-k document references only for carefully chosen indexing term combinations. In addition, since the number of possible term combinations extracted from a document collection can be very large, we propose to use query statistics to index only such combinations that are indeed frequently requested by the users. Thus, by avoiding the maintenance of superfluous indexing information, we achieve a substantial reduction in bandwidth and storage. A specific activation mechanism is applied to continuously update the indexing information according to changes in the query distribution, resulting in an efficient, constantly evolving query-driven indexing structure. We show that the size of the index and the generated indexing/retrieval traffic remains manageable even for websize document collections at the price of a marginal loss in precision for rare queries. Our theoretical analysis and experimental results provide convincing evidence about the feasibility of the query-driven indexing strategy for large scale P2P text retrieval. Moreover, our experiments confirm that the retrieval performance is only slightly lower than the one obtained with state-of-the-art centralized query engines.

PCIR: Combining DHTs and peer clusters for efficient full-text P2P indexing

“…However, PCIR can also efficiently be used to optimize document-granularity indexing [4] as well as query-driven indexing [13,14], as we show in Section 6.…”

“…For example, in a recent version of ALVIS [14], peers identify frequent multi-term queries, and cache their results in the DHT. This offers faster query execution with lower network overhead, albeit at the expense of a larger inverted index over the DHT.…”

Peer-to-peer information retrieval

“…This offers a solution to one of the main drawbacks of using distributed hash tables: intersection of large posting lists. Cuenca-Acuna et al, 2003;Suel et al, 2003;Tang and Dwarkadas, 2004;Balke et al, 2005;Michel et al, 2005a;Zhang and Suel, 2005;Skobeltsyn and Aberer, 2006;Skobeltsyn et al, 2007aSkobeltsyn et al, , 2009 Processing only a subset of items during the search process can yield performance benefits: less data processing and lower latency. Various algorithms, discussed shortly, can be used to retrieve the top items for a particular query without having to calculate the scores for all the items.…”

“…Hence, the topology of the network is not determined by a key space. (Reynolds and Vahdat, 2003;Skobeltsyn and Aberer, 2006;Skobeltsyn et al, 2007a;Zimmer et al, 2008;Skobeltsyn et al, 2007bSkobeltsyn et al, , 2009 It makes little sense to reconstruct the search result set for the same query over and over again if it does not really change. Performance can be increased significantly by caching search results.…”

“…Michel et al (2006) show that MIPs are much more accurate than Bloom filters for this type of estimation. (Skobeltsyn et al, 2009;Luu et al, 2006) An alternative way of reducing the costs of posting list intersection for a distributed global index is by making the lists themselves shorter. To achieve this, instead of building an index over single terms, one can build one over entire multiterm queries.…”