Strategies for building distributed information retrieval systems

Macleod, Ian A.; Martin, Patrick; Nordin, Brent; Phillips, John R.

doi:10.1016/0306-4573(87)90056-2

Cited by 16 publications

(8 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Early studies on distributed indexes for information retrieval systems appeared in [2,11,16,9,21,12,19]. In [2,11,16,9,21,12] various data partitioning schemes among servers are discussed.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Basic issues on the processing of web queries

Badué

Barbosa

Golgher

et al. 2005

Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Search engines represent a key component of Web economy these days. Despite that, there is not much technical literature available on their design, fine tuning, and internal operation. In this work, we make a preliminary attempt to partially fulfill this gap. We distinguish that Web query processing is composed of two phases: (a) retrieving information on documents related to the queries and ranking them, and (b) generating snippets, title, and URL information for the answer page. The second phase has cost that is basically constant on the size of the collection, while the cost of the first phase is affected by the size of the collection. Thus, we concentrate here on studying the behavior of a search engine while executing the first phase of query processing. Using real data and a small cluster of index servers, we study four basic and key issues related to this first phase of query processing: load balance, broker behavior, performance by individual index servers, and overall throughput. Our study, while preliminary, does reveal interesting tradeoffs: (1) that load unbalance at low query arrival rates can be controlled with a simple measure of randomizing the distribution of documents among the index servers, (2) that the broker is not a bottleneck, (3) that disk and CPU utilization at individual servers depends on the relationship between memory size and the distribution of frequencies for the query terms, and (4) that load unbalance at high loads prevents higher throughput. Our results suggest that further studying and evaluating search engines is a promising research avenue.

show abstract

Section: Related Workmentioning

confidence: 99%

“…In [2,11,16,9,21,12] various data partitioning schemes among servers are discussed. This was an issue of early concern.…”

Section: Related Workmentioning

confidence: 99%

Basic issues on the processing of web queries

Badué

Barbosa

Golgher

et al. 2005

Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…A number of studies have investigated the performance of distributed IR systems [5,6,9,12,20,25]. Most of the previous work experiments with a text database less than 1 GB and focuses on speedup when a text database is distributed over more servers [5,12,17,20].…”

Section: Scalable Ir Architecturesmentioning

confidence: 99%

“…Most of the previous work experiments with a text database less than 1 GB and focuses on speedup when a text database is distributed over more servers [5,12,17,20]. Only Couvreur et al [9], and Cahoon et al [6,7] use simulation to experiment with more than 100 GB of data.…”

Section: Scalable Ir Architecturesmentioning

confidence: 99%

Partial collection replication versus caching for information retrieval systems

Lu¹,

McKinley

2000

Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

The explosion of content in distributed information retrieval (IR) systems requires new mechanisms to attain timely and accurate retrieval of unstructured text. In this paper, we compare two mechanisms to improve IR system performance: partial collection replication and caching. When queries have locality, both mechanisms return results more quickly than sending queries to the original collection(s). Caches return results when queries exactly match a previous one. Partial replicas are a form of caching that return results when the IR technology determines the query is a good match. Caches are simpler and faster, but replicas can increase locality by detecting similarity between queries that are not exactly the same. We use real traces from THOMAS and Excite to measure query locality and similarity. With a very restrictive definition of query similarity, similarity improves query locality up to 15% over exact match. We use a validated simulator to compare their performance, and find that even if the partial replica hit rate increases only 3 to 6%, it will outperform simple caching under a variety of configurations. A combined approach will probably yield the best performance.

show abstract

“…We require that each of the subcollections can be accessed without recourse to any central information; and that any subcollection can be a logical component of databases managed by several different receptionists. Note that this independence of librarians eliminates some other possible models for distribution, such as having component files (for example, text, index and dictionary files) at different sites [11] or partitioning the index so that different terms are indexed at different sites [19].…”

Section: Distributed Retrievalmentioning

confidence: 99%

Methodologies for distributed information retrieval

Kretser

Moffat

Shimmin³

et al.

Proceedings. 18th International Conference on Distributed Computing Systems (Cat. No.98CB36183)

View full text Add to dashboard Cite

Text collections have traditionally been located at a single site and managed as a monolithic whole. However, it is now common for a collection to be spread over several hosts and for these hosts to be geographically separated. In this paper we examine several alternative approaches to distributed text retrieval. We report on our experience with a full implementation of these methods, and give retrieval efficiency and retrieval effectiveness results for collections distributed over both a local area network and a wide area network. We conclude that, compared to monolithic systems, distributed information retrieval systems can be fast and effective, but that they are not efficient.

show abstract

Strategies for building distributed information retrieval systems

Cited by 16 publications

References 2 publications

Basic issues on the processing of web queries

Basic issues on the processing of web queries

Partial collection replication versus caching for information retrieval systems

Methodologies for distributed information retrieval

Contact Info

Product

Resources

About