Stratified Sampling for Data Mining on the Deep Web

Liu, Tantan; Wang, Fan; Agrawal, Gagan

doi:10.1109/icdm.2010.17

Cited by 19 publications

(12 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[7][8][9] describe efficient techniques to obtain random samples from hidden web databases that can then be utilized to perform aggregate estimation. Recent works such as [16,24] propose more sophisticated sampling techniques that reduce variance of aggregate estimation. Skyline Computation: Skyline operator was first described in [4] and number of subsequent work have studied it from diverse contexts.…”

Section: Related Workmentioning

confidence: 99%

Discovering the skyline of web databases

et al. 2016

View full text Add to dashboard Cite

Many web databases are "hidden" behind proprietary search interfaces that enforce the top-k output constraint, i.e., each query returns at most k of all matching tuples, preferentially selected and returned according to a proprietary ranking function. In this paper, we initiate research into the novel problem of skyline discovery over top-k hidden web databases. Since skyline tuples provide critical insights into the database and include the top-ranked tuple for every possible ranking function following the monotonic order of attribute values, skyline discovery from a hidden web database can enable a wide variety of innovative third-party applications over one or multiple web databases. Our research in the paper shows that the critical factor affecting the cost of skyline discovery is the type of search interface controls provided by the website. As such, we develop efficient algorithms for three most popular types, i.e., one-ended range, free range and point predicates, and then combine them to support web databases that feature a mixture of these types. Rigorous theoretical analysis and extensive real-world online and offline experiments demonstrate the effectiveness of our proposed techniques and their superiority over baseline solutions.

show abstract

Section: Related Workmentioning

confidence: 99%

Discovering the skyline of web databases

et al. 2016

View full text Add to dashboard Cite

show abstract

“…However, these researches focus on how to build an interactive query system or a vertical search system using data integration technologies [16–18]. Recently, with the development of sampling and crawling over the deep web [19–21], mining deep web has attracted more attention than before [22–24]. Moreover, outlier detection research has always been a hot topic in machine learning and data mining.…”

Section: Related Workmentioning

confidence: 99%

Stratification-Based Outlier Detection over the Deep Web

Xian

Sheng

Fang

et al. 2016

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

For many applications, finding rare instances or outliers can be more interesting than finding common patterns. Existing work in outlier detection never considers the context of deep web. In this paper, we argue that, for many scenarios, it is more meaningful to detect outliers over deep web. In the context of deep web, users must submit queries through a query interface to retrieve corresponding data. Therefore, traditional data mining methods cannot be directly applied. The primary contribution of this paper is to develop a new data mining method for outlier detection over deep web. In our approach, the query space of a deep web data source is stratified based on a pilot sample. Neighborhood sampling and uncertainty sampling are developed in this paper with the goal of improving recall and precision based on stratification. Finally, a careful performance evaluation of our algorithm confirms that our approach can effectively detect outliers in deep web.

show abstract

“…[1] proposed an adaptive sampling algorithm for answering aggregation queries over websites with hierarchical structure. Recent works such as [25,31] propose more sophisticated sampling techniques so as to reduce the variance of the aggregate estimation. For hidden databases with keyword interfaces, prior work have studied estimating the size of search engines [5,4,32], corpus [6] or document collection [29].…”

Section: Related Work Information Integration and Extraction For Hiddmentioning

confidence: 99%

Aggregate estimation over dynamic hidden web databases

et al. 2014

View full text Add to dashboard Cite

Many databases on the web are "hidden" behind (i.e., accessible only through) their restrictive, form-like, search interfaces. Recent studies have shown that it is possible to estimate aggregate query answers over such hidden web databases by issuing a small number of carefully designed search queries through the restrictive web interface. A problem with these existing work, however, is that they all assume the underlying database to be static, while most realworld web databases (e.g., Amazon, eBay) are frequently updated. In this paper, we study the novel problem of estimating/tracking aggregates over dynamic hidden web databases while adhering to the stringent query-cost limitation they enforce (e.g., at most 1,000 search queries per day). Theoretical analysis and extensive realworld experiments demonstrate the effectiveness of our proposed algorithms and their superiority over baseline solutions (e.g., the repeated execution of algorithms designed for static web databases).

show abstract

Stratified Sampling for Data Mining on the Deep Web

Cited by 19 publications

References 18 publications

Discovering the skyline of web databases

Discovering the skyline of web databases

Stratification-Based Outlier Detection over the Deep Web

Aggregate estimation over dynamic hidden web databases

Contact Info

Product

Resources

About