Extracting Data behind Web Forms

Abstract

A significant and ever-increasing amount of data is accessible only by filling out HTML forms to query an underlying Web data source. While this is most welcome from a user perspective (queries are relatively easy and precise) and from a data management perspective (static pages need not be maintained and databases can be accessed directly), automated agents must face the challenge of obtaining the data behind forms. In principle an agent can obtain all the data behind a form by multiple submissions of the form filled out in all possible ways, but efficiency concerns lead us to consider alternatives. We investigate these alternatives and show that we can estimate the amount of remaining data (if any) after a small number of submissions and that we can heuristically select a reasonably minimal number of submissions to maximize the coverage of the data. Experimental results show that these statistical predictions are appropriate and useful.

Semantic Enterprise Search (but no Web 2.0)

“…We also see an apparent solution to the issues explained so far in the "Semantic Search" research field 4 . There is a handful of related and distinct approaches, some of which we will introduce here:…”

Automating the extraction of data from HTML tables with unknown structure

“…Thus, in order to arrive at much of the data we can process with the system we have proposed in this paper, we need to access the hidden Web, a problem on which we are currently working [LYE01,LESY02]. Once extracted, if the result is a we also plan to piece together all the components we have developed in our data-extraction work [DEG] into a comprehensive extraction tool.…”

Generating Queries to Crawl Hidden Web Using Keyword Sampling and Random Forest Classifier

“…In [9] the system is very efficient as it automatically generates new queries from the result of the previous queries but in this crawler the system is not properly indexed.…”