Shard ranking and cutoff estimation for topically partitioned collections

Abstract

Large document collections can be partitioned into topical shards to facilitate distributed search [19]. In a low-resource search environment only a few of the shards can be searched in parallel. Such a search environment faces two intertwined challenges. First, determining which shards to consult for a given query: shard ranking. Second, how many shards to consult from the ranking: cutoff estimation. In this paper we present a family of three algorithms that address both of these problems. As a basis we employ a commonly used data structure, the central sample index (CSI) [29], to represent the shard contents. Running a query against the CSI yields a flat document ranking that each of our algorithms transforms into a tree structure. A bottom up traversal of the tree is used to infer a ranking of shards and also to estimate a stopping point in this ranking that yields cost-effective selective distributed search. As compared to a state-of-theart shard ranking approach the proposed algorithms provide substantially higher search efficiency while providing comparable search effectiveness.

Statistical comparisons of non-deterministic IR systems using two dimensional variance

“…There are several other resource allocation techniques available but covering all of them is beyond the scope of this paper. See for example Aly et al (2013), Callan, Lu, and Croft (1995), Kulkarni, Tigelaar, Hiemstra, and Callan (2012), and Xu and Callan (1998). Here we focus on resource allocation schemes which depend on sampling, such as ReDDE and Rank-S. We refer to any sample-based resource allocation scheme as a central sample index (CSI) in the remainder of this paper.…”

Rank-energy selective query forwarding for distributed search systems

“…Distributed information retrieval (IR) is of substantial interest to the IR research community (e.g., [2][3][4]6], [8], [9], [23], [24], and [26][27][28]). Although traditionally focused on efficiency within a search site, such as by partitioning a document collection into smaller collections or "shards" (e.g., [26][27][28]), recent research has demonstrated the feasibility of distributed IR at the system level (e.g., [2], [4], [8], [9], [23], and [24]).…”

“…Although traditionally focused on efficiency within a search site, such as by partitioning a document collection into smaller collections or "shards" (e.g., [26][27][28]), recent research has demonstrated the feasibility of distributed IR at the system level (e.g., [2], [4], [8], [9], [23], and [24]). …”

“…Request permissions from permissions@acm.org. CIKM'13, Oct. 27 Figure 1: Energy price temporal and spatial variation: (top) NYISO energy prices [29], 15 minute intervals; (bottom) EIA energy prices by country [37].…”

Reducing hardware hit by queries in web search engines

“…Kulkarni, Tigelaar, Hiemstra, and Callan (2012) proposed three algorithms in the context of topic-based partitions to decide which collection and how many partitions to select. It used the so called central sample index (CSI) as an inverted index of a small sample of randomly selected documents from each partition.…”