Efficiently Computing Inclusion Dependencies for Schema Discovery

Abstract

Large data integration projects must often cope with undocumented data sources. Schema discovery aims at automatically finding structures in such cases. An important class of relationships between attributes that can be detected automatically are inclusion dependencies (IND), which provide an excellent basis for guessing foreign key constraints. INDs can be discovered by comparing the sets of distinct values of pairs of attributes.In this paper we present efficient algorithms for finding unary INDs. We first show that (and why) SQL is not suitable for this task. We then develop two algorithms that compute inclusion dependencies outside of the database. Both are much faster than the SQL-based methods; in fact, for larger schemas they are the only feasible solution. Our experiments show that we can compute all unary INDs in a schema of 1, 680 attributes with a total database size of 3.2 GB in approximately 2.5 hours.

Efficient filtering and ranking schemes for finding inclusion dependencies on the web

“…Therefore, many studies have addressed the problem of helping users find integrity constraints from an existing data instance. However, most existing techniques address the problem of supporting the discovery of data integrity constraints in the context of relational databases [1] [2]. To the best of our knowledge, only a few papers address the problem of supporting the discovery of data integrity constraints in the Web context.…”

“…To the best of our knowledge, the complexity of the fastest algorithm to check if a pair (e i , e j ) is an inclusion is in O(n) for the size of the sets [2] under the assumption that we sort the words in e i and e j before the calculation.…”

“…Bauckmann and others [2] proposed an algorithm that takes as input a set of relations and efficiently enumerates all pairs of relational attributes one of which includes the other. The algorithm is designed to minimize the amount of I/O over the sets of attribute values.…”

“…Because it is inevitable that finding all inclusion dependencies in a given data set yields false positives, a common approach is first to enumerate all possible candidates (including false positives) [2] and then to verify the enumerated candidates. This paper discusses algorithms for this two-phase approach in the Web context.…”

“…We want the scheme to be efficient, because the number of Web pages can be large. (2) Dealing with the characteristics of Web content. We want the scheme to be able to deal with the characteristics of Web content, because Web pages have hierarchical structures and their data are not necessarily clean.…”