Extraction and integration of partially overlapping web sources

Abstract

We present an unsupervised approach for harvesting the data exposed by a set of structured and partially overlapping data-intensive web sources. Our proposal comes within a formal framework tackling two problems: the data extraction problem, to generate extraction rules based on the input websites, and the data integration problem, to integrate the extracted data in a unified schema. We introduce an original algorithm, WEIR, to solve the stated problems and formally prove its correctness. WEIR leverages the overlapping data among sources to make better decisions both in the data extraction (by pruning rules that do not lead to redundant information) and in the data integration (by reflecting local properties of a source over the mediated schema). Along the way, we characterize the amount of redundancy needed by our algorithm to produce a solution, and present experimental results to show the benefits of our approach with respect to existing solutions.

Extraction and integration of web sources with humans and domain knowledge

“…An interesting solution to achieve high scalability in extraction and integration is by exploiting the redundancy of published information of multiple web sources [2], or engaging humans to improve the performances [3]. In [2] the authors observe that web sources that publish information about the same domain often show a redundancy at the schema level and a partial overlap at instance level.…”

“…In [2] the authors observe that web sources that publish information about the same domain often show a redundancy at the schema level and a partial overlap at instance level. Aligning instances from different sources provides an automatic technique to address synergically both extraction and integration of data from web sources.…”

“…We envision a framework that combines automatic data extraction and integration techniques [2,8] with a supervised approach [5,6] guided by the crowd.…”

“…Different techniques can be adopted to reduce the number of generated rules and to discard rules that extract non relevant template nodes [2].…”

“…Another attempt to scale the extraction and integration of web data is by relying on partially overlapping web sources [2], web sources that push redundant information at schema level and at instance level. They adopt a "lazy" approach so that the schema of multiple web sources of the same domain is learned during the extraction process, in a bottom up fashion.…”