Improving data quality by source analysis

Abstract

In many domains, data cleaning is hampered by our limited ability to specify a comprehensive set of integrity constraints to assist in identification of erroneous data. An alternative approach to improve data quality is to exploit different data sources that contain information about the same set of objects. Such overlapping sources highlight hot-spots of poor data quality through conflicting data values and immediately provide alternative values for conflict resolution. In order to derive a dataset of high quality, we can merge the overlapping sources based on a quality assessment of the conflicting values. The quality of the resulting dataset, however, is highly dependent on our ability to asses the quality of conflicting values effectively.The main objective of this article is to introduce methods that aid the developer of an integrated system over overlapping, but contradicting sources in the task of improving the quality of data. Value conflicts between contradicting sources are often systematic, caused by some characteristic of the different sources. Our goal is to identify such systematic differences and outline data patterns that occur in conjunction with them. Evaluated by an expert user, the regularities discovered provide insights into possible conflict reasons and help to assess the quality of inconsistent values. The contributions of this article are two concepts of systematic conflicts: contradiction patterns and minimal update sequences. Contradiction patterns resemble a special form of association rules that summarize characteristic data properties for conflict occurrence. We adapt existing association rule mining algorithms for mining contradiction patterns. Contradiction patterns, however, view each class of conflicts in isolation, sometimes leading to largely overlapping patterns. Sequences of set-oriented update operations that transform one data source into the other are compact descriptions for all regular differences among the sources. We consider minimal update sequences as the most likely explanation for observed differences between overlapping data sources. Furthermore, the order of operations within the sequences point out potential dependencies between systematic differences. Finding minimal update sequences, however, is beyond reach in practice. We show that the problem already is NP-complete for a restricted set of operations. In the light of this intractability result, we present heuristics that lead to convincing results for all examples we considered.

Comparative genomics of casein genes

“…Some of these differences are rather extreme, for example, the gene gain or loss tree shows that the squirrel only has β-casein, while the rest of the caseins are absent. It is of paramount importance to note that comparative genomics data is only as good as the quality of the genome databases (Muller et al ., 2003). Sequencing errors that are carried over to the actual genome database may be misleading and therefore result in inaccurate interpretation of results.…”

Developing a Global Data Breach Database and the Challenges Encountered

“…As a result, companies and entities increasingly rely on data to support decision making and gain a competitive advantage. To make informed and effective decisions, it is crucial to evaluate and guarantee the quality of the cybersecurity data [Muller, Freytag, and Leser 2012] Even with the issues listed throughout this study, we were able to establish several criteria and analyses to mitigate possible impacts on the results. Nevertheless, it is necessary to study the best ways to gather, analyze, generate reports, and more trustworthy research regarding data breach incidents.…”

Improving data quality in large-scale repositories through conflict resolution

“…Conflicts are resolved by defining a conflict resolution function, which should be declaratively specified [27]. As discussed by Müller et al [26], "conflicts between contradicting sources are often systematic, caused by some characteristic of the different sources". The goal for experts is to identify these characteristics and use their domain knowledge to resolve conflicts.…”

“…To resolve conflicts efficiently, experts require effective tool support that allows them to explore the data and understand its characteristics, then formulate resolution strategies and apply them to the data set. In doing so, the ability to "exploit different data sources that contain information about the same set of objects" is key [26].…”