Mining functional dependencies from data

Yao, Hong; Hamilton, Howard J.

doi:10.1007/s10618-007-0083-9

Cited by 69 publications

(73 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A functional dependency states that values in one set of columns functionally determine the value of another column. Again, much research has been performed to automatically detect Fds [75,139]. Section 5 surveys dependency discovery algorithms in detail.…”

Section: Dependenciesmentioning

confidence: 99%

“…TANE [75], FUN [110], and FD_Mine [139] are three algorithms that follow this strategy, with FUN and FD_Mine introducing additional pruning rules beyond TANE's based on the properties of Fds. They start with sets of single columns in the LHS and work their way up the powerset lattice in a level-wise manner.…”

Section: Column-based Algorithmsmentioning

confidence: 99%

“…For instance, if we find that A → B and B → A, then we can prune all LHS column sets including B, because A and B are equivalent [139]. Another pruning strategy is to ignore columns sets that have the same number of distinct values as their subsets [110].…”

Section: Column-based Algorithmsmentioning

confidence: 99%

See 2 more Smart Citations

Profiling relational data: a survey

Abedjan¹,

2015

View full text Add to dashboard Cite

Profiling data to determine metadata about a given dataset is an important and frequent activity of any IT professional and researcher and is necessary for various use-cases. It encompasses a vast array of methods to examine datasets and produce metadata. Among the simpler results are statistics, such as the number of null values and distinct values in a column, its data type, or the most frequent patterns of its data values. Metadata that are more difficult to compute involve multiple columns, namely correlations, unique column combinations, functional dependencies, and inclusion dependencies. Further techniques detect conditional properties of the dataset at hand. This survey provides a classification of data profiling tasks and comprehensively reviews the state of the art for each class. In addition, we review data profiling tools and systems from research and industry. We conclude with an outlook on the future of data profiling beyond traditional profiling tasks and beyond relational databases.

show abstract

Section: Dependenciesmentioning

confidence: 99%

Section: Column-based Algorithmsmentioning

confidence: 99%

See 1 more Smart Citation

Profiling relational data: a survey

Abedjan¹,

2015

View full text Add to dashboard Cite

show abstract

“…Another form of dependency, which is also relevant for data quality, is the functional dependency (FD). Again, much research has been performed to automatically detect FDs [26,45].…”

Section: State Of the Artmentioning

confidence: 99%

Data profiling revisited

Naumann

2014

SIGMOD Rec.

129

View full text Add to dashboard Cite

Data profiling comprises a broad range of methods to efficiently analyze a given data set. In a typical scenario, which mirrors the capabilities of commercial data profiling tools, tables of a relational database are scanned to derive metadata, such as data types and value patterns, completeness and uniqueness of columns, keys and foreign keys, and occasionally functional dependencies and association rules. Individual research projects have proposed several additional profiling tasks, such as the discovery of inclusion dependencies or conditional functional dependencies.Data profiling deserves a fresh look for two reasons: First, the area itself is neither established nor defined in any principled way, despite significant research activity on individual parts in the past. Second, more and more data beyond the traditional relational databases are being created and beg to be profiled. The article proposes new research directions and challenges, including interactive and incremental profiling and profiling heterogeneous and non-relational data.

show abstract

“…Yet they have been proved not to scale properly for big loads of data, which is the most common scenario in DW systems, as pointed out, for example, in (C. Monash, 2008;Golfarelli and Rizzi, 2009). For this reason, this is still a research topic giving rise to new proposals, like (Sismanis, Brown, Haas and Reinwald, 2006;Yao and Hamilton, 2008;Yeh, Li and Chu, 2008). Furthermore, note that these approaches exclusively work over instances and they cannot easily tolerate erroneous data (that may generate fake IDs that do not really hold or overlook real ones).…”

Section: Related Workmentioning

confidence: 99%

Ontology driven search of compound IDs

Abelló

Romero

2011

Knowl Inf Syst

View full text Add to dashboard Cite

Abstract. Object identification is a crucial step in most information systems. Nowadays, we have many different ways to identify entities such as surrogates, keys and object identifiers. However, not all of them guarantee the entity identity. Many works have been introduced in the literature for discovering meaningful identifiers (i.e., guaranteeing the entity identity according to the semantics of the universe of discourse), but all of them work at the logical or data level and they share some constraints inherent to the kind of approach. Addressing it at the logical level, we may miss some important data dependencies, while the cost to identify data dependencies purely at the data level may not be affordable. In this paper, we propose an approach for discovering meaningful identifiers driven by domain ontologies. In our approach, we guide the process at the conceptual level and we introduce a set of pruning rules for improving the performance by reducing the number of identifier hypotheses generated and to be verified with data. Finally, we also introduce a simulation over a case study to show the feasibility of our method.

show abstract

Mining functional dependencies from data

Cited by 69 publications

References 20 publications

Profiling relational data: a survey

Profiling relational data: a survey

Data profiling revisited

Ontology driven search of compound IDs

Contact Info

Product

Resources

About