Automatic Filling of Hidden Web Forms

Abstract

A significant part of the information available on the Web is stored in online databases which compose what is known as Hidden Web or Deep Web. In order to access information from the Hidden Web, one must fill an HTML form that is submitted as a query to the underlying database. In recent years, many works have focused on how to automate the process of form filling by creating methods for choosing values to fill the fields in the forms. This is a challenging task since forms may contain fields for which there are no predefined values to choose from. This article presents a survey of methods for Web Form Filling, analyzing the existing solutions with respect to the type of forms that they handle and the filling strategy adopted. We provide a comparative analysis of 15 key works in this area and discuss directions for future research.

Deep Web crawling: a survey

“…Finally, Kantorski et al [47] presented the most recent survey on form filling. Their analysis covers 15 proposals from 2001 to 2013.…”

“…Deep Web crawling has a number of applications. The most usual ones are: creating indexed repositories of deep Web pages for search engines [9,25,47] and periodically refreshing the repositories to keep them updated [19,25,71,73], performing automated testing of Web sites [37], or virtual integration, that is, offering a unified view on the data offered by different Web sites [24,87,96].…”

“…To the best of our knowledge, there are four surveys on deep Web crawling [25,61,70,76], but they do not provide a framework that allows comparing the proposals, and they do not analyse the measures and datasets that are used to evaluate the crawlers performance. There are two recent surveys on form filling techniques [47,49], but they do not analyse the other steps in the deep Web crawling process. All these issues have motivated us to work on this survey, in which we propose a framework that analyses the main features of existing deep Web crawling related techniques, including the latest proposals in automated deep Web discovery, form filling, and crawling path learning, and we provide an overall picture regarding deep Web crawling.…”

A New Hidden Web Crawling Approach

“…The public web normally deploys by common use search engine, hidden web represents information, stored in specialized databases, only accessible through specific search interfaces created by using CGI and HTML forms or JavaScript etc. [3] However, a number of recent studies [4,5,6,7] have observed that a significant fraction of Web content in fact lies outside the PIW( Publicly Indexable Web). Specifically, large portions of the Web are "hidden" behind search forms, in searchable structured and unstructured databases (called the hidden Web or deep Web [6]).…”

“…[3] However, a number of recent studies [4,5,6,7] have observed that a significant fraction of Web content in fact lies outside the PIW( Publicly Indexable Web). Specifically, large portions of the Web are "hidden" behind search forms, in searchable structured and unstructured databases (called the hidden Web or deep Web [6]). Pages in the hidden Web are dynamically generated in response to queries submitted via the search forms.…”