A Bayesian approach to discovering truth from conflicting sources for data integration

Abstract

In practical data integration systems, it is common for the data sources being integrated to provide conflicting information about the same entity. Consequently, a major challenge for data integration is to derive the most complete and accurate integrated records from diverse and sometimes conflicting sources. We term this challenge the truth finding problem. We observe that some sources are generally more reliable than others, and therefore a good model of source quality is the key to solving the truth finding problem. In this work, we propose a probabilistic graphical model that can automatically infer true records and source quality without any supervision. In contrast to previous methods, our principled approach leverages a generative process of two types of errors (false positive and false negative) by modeling two different aspects of source quality. In so doing, ours is also the first approach designed to merge multi-valued attribute types. Our method is scalable, due to an efficient sampling-based inference algorithm that needs very few iterations in practice and enjoys linear time complexity, with an even faster incremental variant. Experiments on two real world datasets show that our new method outperforms existing state-ofthe-art approaches to the truth finding problem.

A confidence-aware approach for truth discovery on long-tail data

“…It models the propagation of information trustworthiness from the known ground truths. Zhao et al adopt probabilistic graphical models in truth discovery tasks [36,37]. The existence of multiple truths for single entity is considered in [37] where source reliability is modeled as two-sided: sensitivity and specificity.…”

“…Zhao et al adopt probabilistic graphical models in truth discovery tasks [36,37]. The existence of multiple truths for single entity is considered in [37] where source reliability is modeled as two-sided: sensitivity and specificity. Later, a model specially designed for numerical data is proposed in [36].…”

“…Second, as the majority of sources claims very few facts, the removal of these sources may result in sparse data and limited coverage. An alternative strategy could be drawn from Bayesian estimation in which a smoothing prior can be added [36,37]. We can add a fixed "pseudo" count in the computation of source accuracy so that the estimation can be smoothed for sources with very few claims.…”

A Data Partition Based Model to Enforce Security in Cloud Database

“…Hiding dependencies among data items and denying write access to some data items are the two methods to prevent the attacks. Papers like [10] and [11] propose a probabilistic graphical model that can automatically infer true records and source quality in cloud data without any supervision. They leverage a generative process of two types of errors (false positive and false negative) by modeling two different aspects of source quality.…”

Mining collective intelligence in diverse groups

“…The participating sources collaborate to claim their own observations, such as facts and labels, on these objects. Our goal is to aggregate these collective observations to infer the true values (e.g., the true fact and image label) for the different objects [18,14,5].…”