Approximate Discovery of Functional Dependencies for Large Datasets

Bleifuß, Tobias; Bülow, Susanne; Frohnhofen, Johannes; Risch, Julian; Wiese, Georg; Kruse, Sebastian; Papenbrock, Thorsten; Naumann, Felix

doi:10.1145/2983323.2983781

Cited by 19 publications

(17 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, there is such a functional dependency A->B, which means that any two records in the relationship, when their values on the attribute set A are equal, the values on the attribute set B must be equal. Bleifuß et al [66] propose an approximate discovery strategy AID-FD (Approximate Iterative Discovery of FDs) which sacrifices a certain correct rate in exchange for performance improvement. AID-FD uses an incremental, focused sampling of tuple pairs to deduce non-FDs until user-configured termination criterion is met.…”

Section: B Sampling For Functional Dependenciesmentioning

confidence: 99%

Sampling for Big Data Profiling: A Survey

Liu

Zhang

2020

IEEE Access

View full text Add to dashboard Cite

Due to the development of internet technology and computer science, data is exploding at an exponential rate. Big data brings us new opportunities and challenges. On the one hand, we can analyze and mine big data to discover hidden information and get more potential value. On the other hand, the 5V characteristic of big data, especially Volume which means large amount of data, brings challenges to storage and processing. For some traditional data mining algorithms, machine learning algorithms and data profiling tasks, it is very difficult to handle such a large amount of data. The large amount of data is highly demanding hardware resources and time consuming. Sampling methods can effectively reduce the amount of data and help speed up data processing. Sampling technology has been widely used in big data context. Data profiling is the activity that finds metadata of data set and has many use cases, e.g., performing data profiling tasks on relational data, graph data, and time series data for anomaly detection and data repair. However, data profiling is computationally expensive, especially for large data sets. Hence this article focuses on researching sampling for data profiling tasks in big data context and investigates the application of sampling in different categories of data profiling. From the experimental results of these studies, the results got from the sampled data are close to or even exceed the results of the full amount of data. Therefore, sampling technology plays an important role in the era of big data, and we also have reason to believe that sampling technology will become an indispensable step in big data processing in the future.INDEX TERMS Big data, large amount, sampling, data profiling.

show abstract

Section: B Sampling For Functional Dependenciesmentioning

confidence: 99%

Sampling for Big Data Profiling: A Survey

Liu

Zhang

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Pyro focuses on approximate dependencies that may be violated by a certain portion of tuples or tuple pairs. Note that this is different from dependency approximation algorithms [8,22], which trade correctness guarantees of the discovered dependencies for performance improvements. In the following, we focus on those works that share goals or have technical commonalities with Pyro.…”

Section: Related Workmentioning

confidence: 95%

Efficient discovery of approximate dependencies

Kruse

Naumann

2018

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

Functional dependencies (FDs) and unique column combinations (UCCs) form a valuable ingredient for many data management tasks, such as data cleaning, schema recovery, and query optimization. Because these dependencies are unknown in most scenarios, their automatic discovery has been well researched. However, existing methods mostly discover only exact dependencies, i.e., those without violations. Realworld dependencies, in contrast, are frequently approximate due to data exceptions, ambiguities, or data errors. This relaxation to approximate dependencies renders their discovery an even harder task than the already challenging exact dependency discovery. To this end, we propose the novel and highly efficient algorithm Pyro to discover both approximate FDs and approximate UCCs. Pyro combines a separate-and-conquer search strategy with sampling-based guidance that quickly detects dependency candidates and verifies them. In our broad experimental evaluation, Pyro outperforms existing discovery algorithms by a factor of up to 33, scales to larger datasets, and at the same time requires the least main memory.

show abstract

“…There has been a number of different formalisms to extend FDs to handle erroneous data inherent in real-world applications. Common approaches include approximate FDs [5] and conditional FDs [20,21] whereby a FD holds on a subset of data instead of the entire dataset (please refer to [7] for a detail survey of relaxed definitions of FDs). There has been some work on probabilistic FDs that might hold on the data with some probability [10,34].…”

Section: Related Workmentioning

confidence: 99%

Discovery of genuine functional dependencies from relational data with missing values

et al. 2018

Self Cite

View full text Add to dashboard Cite

Functional dependencies (FDs) play an important role in maintaining data quality. They can be used to enforce data consistency and to guide repairs over a database. In this work, we investigate the problem of missing values and its impact on FD discovery. When using existing FD discovery algorithms, some genuine FDs could not be detected precisely due to missing values or some non-genuine FDs can be discovered even though they are caused by missing values with a certain NULL semantics. We define a notion of genuineness and propose algorithms to compute the genuineness score of a discovered FD. This can be used to identify the genuine FDs among the set of all valid dependencies that hold on the data. We evaluate the quality of our method over various real-world and semi-synthetic datasets with extensive experiments. The results show that our method performs well for relatively large FD sets and is able to accurately capture genuine FDs.

show abstract

Approximate Discovery of Functional Dependencies for Large Datasets

Cited by 19 publications

References 13 publications

Sampling for Big Data Profiling: A Survey

Sampling for Big Data Profiling: A Survey

Efficient discovery of approximate dependencies

Discovery of genuine functional dependencies from relational data with missing values

Contact Info

Product

Resources

About