Capturing collection size for distributed non-cooperative retrieval

Shokouhi, Milad; Zobel, Justin; Scholer, Falk; Tahaghoghi, S. M. M.

doi:10.1145/1148170.1148227

Cited by 48 publications

(69 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data Analytics over Hidden Databases: There has been prior work on crawling, sampling, and aggregate estimation over the hidden web, specifically over text [3,4] and structured [16] hidden databases and search engines [15,18,2]. Specifically, samplingbased methods were used for generating content summaries [6,14,11], processing top-k queries [5], etc.…”

Section: Related Workmentioning

confidence: 99%

Rank discovery from web databases

2013

View full text Add to dashboard Cite

Many web databases are only accessible through a proprietary search interface which allows users to form a query by entering the desired values for a few attributes. After receiving a query, the system returns the top-k matching tuples according to a pre-determined ranking function. Since the rank of a tuple largely determines the attention it receives from website users, ranking information for any tuple -not just the top-ranked ones -is often of significant interest to third parties such as sellers, customers, market researchers and investors. In this paper, we define a novel problem of rank discovery over hidden web databases. We introduce a taxonomy of ranking functions, and show that different types of ranking functions require fundamentally different approaches for rank discovery. Our technical contributions include principled and efficient randomized algorithms for estimating the rank of a given tuple, as well as negative results which demonstrate the inefficiency of any deterministic algorithm. We show extensive experimental results over real-world databases, including an online experiment at Amazon.com, which illustrates the effectiveness of our proposed techniques.

show abstract

Section: Related Workmentioning

confidence: 99%

Rank discovery from web databases

2013

View full text Add to dashboard Cite

show abstract

“…This process is repeated and the estimation is based on the size of the number of documents collected in the two samples. Shokouhi et al [18] explores the problem of resource selection in non-cooperative environments and propose a querybased sampling QBS method based on the capture-recapture method [19] referred as multiple capture-recapture. They showed with a large set of heterogeneous collections that their method outperforms previous approaches using search engines in non-cooperative environments.…”

Section: Vertical Size Estimationmentioning

confidence: 99%

“…The sample was constructed by choosing randomly 5 queries from the set of 2K queries (with replacement) and collecting the top 10 results for each query. Similar parameter values have been used in previous studies [18]. It is important to mention that the set of 90 queries employed to gather user assessments were not employed in the samples generated for the size estimation process to avoid bias in the evaluation of vertical selection methods.…”

Section: Vertical Size Estimationmentioning

confidence: 99%

Vertical selection in the information domain of children

Torres

Hiemstra

Huibers

2013

Proceedings of the 13th ACM/IEEE-CS Joint Conference on Digital Libraries

View full text Add to dashboard Cite

In this paper we explore the vertical selection methods in aggregated search in the specific domain of topics for children between 7 and 12 years old. A test collection consisting of 25 verticals, 3.8K queries and relevant assessments for a large sample of these queries mapping relevant verticals to queries was built. We gather relevant assessment by envisaging two aggregated search systems: one in which the Web vertical is always displayed and in which each vertical is assessed independently from the web vertical. We show that both approaches lead to a di↵erent set of relevant verticals and that the former is prone to bias of visually oriented verticals. In the second part of this paper we estimate the size of the verticals for the target domain. We show that employing the global size and domain specific size estimation of the verticals lead to significant improvements when using state-of-the art methods of vertical selection. We also introduce a novel vertical and query representation based on tags from social media and we show that its use lead to significant performance gains.

show abstract

“…Capture-recapture methods (Shokouhi et al 2006) rely on using the overlap between datasets in order to estimate the overall properties of the sampled set. However, in our case, we are sampling truly interacting and falsepositive protein pairs from the possible set of distinct protein pairs: i.e.…”

Section: Experimental Protein Interaction Datamentioning

confidence: 99%

“…the number of items recaptured) is used to estimate the complete population's size. Multiple capture-recapture (Shokouhi et al 2006) is an extension of this approach to account for any number of samples. This has been used to estimate the size of different populations by observing the overlap between different samples (Xu et al 2007).…”

Section: Single Urn Models For Protein Interaction Datamentioning

confidence: 99%

Assessing Coverage of Protein Interaction Data Using Capture–Recapture Models

Kelly

Stumpf

2011

Bull Math Biol

View full text Add to dashboard Cite

Protein interaction networks comprise thousands of individual binary links between distinct proteins. Whilst these data have attracted considerable attention and been the focus of many different studies, the networks, their structure, function, and how they change over time are still not fully known. More importantly, there is still considerable uncertainty regarding their size, and the quality of the available data continues to be questioned. Here, we employ statistical models of the experimental sampling process, in particular capture-recapture methods, in order to assess the false discovery rate and size of protein interaction networks. We uses these methods to gauge the ability of different experimental systems to find the true binary interactome. Our model allows us to obtain estimates for the size and false-discovery rate from simple considerations regarding the number of repeatedly interactions, and provides suggestions as to how we can exploit this information in order to reduce the effects of noise in such data. In particular our approach does not require a reference dataset. We estimate that approximately more than half of the true physical interactome has now been sampled in yeast.

show abstract

Capturing collection size for distributed non-cooperative retrieval

Cited by 48 publications

References 24 publications

Rank discovery from web databases

Rank discovery from web databases

Vertical selection in the information domain of children

Assessing Coverage of Protein Interaction Data Using Capture–Recapture Models

Contact Info

Product

Resources

About