Crawling the Content Hidden Behind Web Forms

Álvarez, Manuel; Raposo, Juan; Pan, Alberto; Cacheda, Fidel; Bellas, Francisco; Carneiro, Victor

doi:10.1007/978-3-540-74477-1_31

Cited by 27 publications

(34 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As far as we know, RoadRunner is the only automatic web data extraction system available for download. 2 Compared to our system, RoadRunner performs neither the region location stage nor the record division stage. Its function is comparable to the stage in our approach which extracts the individual attributes from each data record.…”

Section: Comparison With Roadrunnermentioning

confidence: 98%

“…3. We define the column similarity between t i and t j , denoted cs(t i , t j ), as the inverse of the average absolute error between the columns corresponding to t i and t j in the similarity matrix (2). Therefore, to consider two subtrees as similar, the column similarity measure requires their columns in the similarity matrix to be very similar.…”

Section: Grouping the Subtreesmentioning

confidence: 99%

“…Several works [2,8,15,21,25] have addressed the problem of how to automatically interpret and fill in web query forms to obtain the response pages. [10] has addressed the problem of examining a web site to automatically collect pages following the same template.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Finding and Extracting Data Records from Web Pages

Álvarez

Pan

Raposo

et al. 2008

J Sign Process Syst Sign Image Video Technol

Self Cite

View full text Add to dashboard Cite

Many HTML pages are generated by software programs by querying some underlying databases and then filling in a template with the data. In these situations the metainformation about the data structure is lost, so automated software programs cannot process these data in such powerful manners as information from databases. We propose a set of novel techniques for detecting structured records in a web page and extracting the data values that constitute them. Our method needs only an input page. It starts by identifying the data region of interest in the page. Then it is partitioned into records by using a clustering method that groups similar subtrees in the DOM tree of the page. Finally, the attributes of the data records are extracted by using a method based on multiple string alignment. We have tested our techniques with a high number of real web sources, obtaining high precision and recall values.

show abstract

Section: Comparison With Roadrunnermentioning

confidence: 98%

Section: Grouping the Subtreesmentioning

confidence: 99%

See 1 more Smart Citation

Finding and Extracting Data Records from Web Pages

Álvarez

Pan

Raposo

et al. 2008

J Sign Process Syst Sign Image Video Technol

Self Cite

View full text Add to dashboard Cite

show abstract

“…No page down is allowed. 2 http://www.amazon.com/Best-Sellers/zgbs may prefer a small k to (1) speed up query processing and shorten the returned webpage, and/or (2) thwart web/tuple scraping. However, in order to accommodate the needs of website users, the value of k should not be too small.…”

Section: Introduction a Problem Motivationmentioning

confidence: 99%

Breaking the top-k barrier of hidden web databases?

Thirumuruganathan

Zhang²,

Das

2013

2013 IEEE 29th International Conference on Data Engineering (ICDE)

View full text Add to dashboard Cite

A large number of web databases are only accessible through proprietary form-like interfaces which require users to query the system by entering desired values for a few attributes. A key restriction enforced by such an interface is the top-k output constraint -i.e., when there are a large number of matching tuples, only a few (top-k) of them are preferentially selected and returned by the website, often according to a proprietary ranking function. Since most web database owners set k to be a small value, the top-k output constraint prevents many interesting third-party (e.g., mashup) services from being developed over real-world web databases. In this paper we consider the novel problem of "digging deeper" into such web databases. Our main contribution is the meta-algorithm GetNext that can retrieve the next ranked tuple from the hidden web database using only the restrictive interface of a web database without any prior knowledge of its ranking function. This algorithm can then be called iteratively to retrieve as many top ranked tuples as necessary. We develop principled and efficient algorithms that are based on generating and executing multiple reformulated queries and inferring the next ranked tuple from their returned results. We provide theoretical analysis of our algorithms, as well as extensive experimental results over synthetic and real-world databases that illustrate the effectiveness of our techniques.

show abstract

“…Several papers have dealt with the problem of crawling and downloading information present in hidden text based databases [10]- [12]. [13]- [15] deal with extracting data from structured hidden databases. [16] and [17] use query based sampling methods to generate content summaries with relative and absolute frequencies while [18], [19] uses two phase sampling method on text based interfaces.…”

Section: Analysis Of Alert-hybridmentioning

confidence: 99%

Leveraging COUNT Information in Sampling Hidden Databases

Dasgupta

Zhang

Das

2009

2009 IEEE 25th International Conference on Data Engineering

View full text Add to dashboard Cite

Abstract-A large number of online databases are hidden behind form-like interfaces which allow users to execute search queries by specifying selection conditions in the interface. Most of these interfaces return restricted answers (e.g., only top-k of the selected tuples), while many of them also accompany each answer with the COUNT of the selected tuples. In this paper, we propose techniques which leverage the COUNT information to efficiently acquire unbiased samples of the hidden database. We also discuss variants for interfaces which do not provide COUNT information. We conduct extensive experiments to illustrate the efficiency and accuracy of our techniques.

show abstract

Crawling the Content Hidden Behind Web Forms

Cited by 27 publications

References 10 publications

Finding and Extracting Data Records from Web Pages

Finding and Extracting Data Records from Web Pages

Breaking the top-k barrier of hidden web databases?

Leveraging COUNT Information in Sampling Hidden Databases

Contact Info

Product

Resources

About