The ontological key: automatically understanding and integrating forms to access the deep Web

Abstract

Forms are our gates to the web. They enable us to access the deep content of web sites. Automatic form understanding provides applications, ranging from crawlers over meta-search engines to service integrators, with a key to this content. Yet, it has received little attention other than as component in specific applications such as crawlers or meta-search engines. No comprehensive approach to form understanding exists, let alone one that produces rich models for semantic services or integration with linked open data.In this paper, we present OPAL, the first comprehensive approach to form understanding and integration. We identify form labeling and form interpretation as the two main tasks involved in form understanding. On both problems OPAL pushes the state of the art: For form labeling, it combines features from the text, structure, and visual rendering of a web page. In extensive experiments on the ICQ and TEL-8 benchmarks and a set of 200 modern web forms OPAL outperforms previous approaches for form labeling by a significant margin. For form interpretation, OPAL uses a schema (or ontology) of forms in a given domain. Thanks to this domain schema, it is able to produce nearly perfect (> 97% accuracy in the evaluation domains) form interpretations. Yet, the effort to produce a domain schema is very low, as we provide a Datalog-based template language that eases the specification of such schemata and a methodology for deriving a domain schema largely automatically from an existing domain ontology. We demonstrate the value of OPAL's form interpretations through a light-weight form integration system that successfully translates and distributes master queries to hundreds of forms with no error, yet is implemented with only a handful translation rules.

ROSeAnn

“…Also, the ability of ROSEANN to annotate different sections of the DOM with different annotation pools, together with its reconciliation capabilities, reduce the noise in the annotations that is the main source of errors in annotationdriven wrapper inducers such as [3]. Figure 6 shows the use of ROSEANN within DIADEM, in particular, for the unsupervised segmentation of classified listings on the web [6] and understanding of forms [5].…”

VADA: an architecture for end user informed data preparation

“…The data wrangling functionality, for example for mapping generation or format transformation, is implemented as a collection of loosely coupled components that build on the concept of a relational transducer (or simply transducer). Transducers were introduced by Abiteboul et al [5], and have been successfully applied and extended in a variety of applications [10], including web data extraction [11].…”

“…• δ guard [11] are Vadalog rules that describe whether a transducer is ready to be executed. • δ scopes [11] are Vadalog rules that describe the scope of the transducer (i.e., parts of the knowledge base and external sources that the transducers depend on). • δ map are Vadalog rules that describe the mapping between the knowledge base and other external schemata, and the internal schema of the transducer.…”

Discovering the Deep Web through XML Schema Extraction

“…Another interesting method in the same family is the Ontology-based web Pattern Analysis with Logic (OPAL) method (Furche, 2011) (Furche, 2013). This method does not take in entry a set of deep web sources URL but only the domain name of interest.…”