Clean Answers over Dirty Databases: A Probabilistic Approach

Andritsos, Periklis; Fuxman, Ariel; Miller, Renée J.

doi:10.1109/icde.2006.35

Cited by 133 publications

(157 citation statements)

References 19 publications

Supporting

Mentioning

154

Contrasting

Unclassified

Order By: Relevance

“…The issue of querying and representing such probabilistic data has received much attention in recent years [4,24,8,44,11,2,39], and some of the challenges we face form active research focuses in that area. We plan to utilize the techniques developed in that work to a large extent in building our system, especially language extensions for querying such tables.…”

Section: Dpms As Database Viewsmentioning

confidence: 99%

“…Several research efforts are underway to build systems to manage uncertain data (e.g. MYSTIQ [11], Trio [44], ORION [8], ConQuer [2]). As we discussed in Section 3, views based on dynamic probabilistic models are naturally probabilistic, and we plan to use the techniques developed in the probabilistic databases research, especially query languages and semantics, in our future work.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Online Filtering, Smoothing and Probabilistic Modeling of Streaming data

Kanagal

Deshpande

2008

2008 IEEE 24th International Conference on Data Engineering

116

View full text Add to dashboard Cite

In this paper, we address the problem of extending a relational database system to facilitate efficient real-time application of dynamic probabilistic models to streaming data. We use the recently proposed abstraction of model-based views for this purpose, by allowing users to declaratively specify the model to be applied, and by presenting the output of the models to the user as a probabilistic database view. We support declarative querying over such views using an extended version of SQL that allows for querying probabilistic data. Underneath we use particle filters, a class of sequential Monte Carlo algorithms commonly used to implement dynamic probabilistic models, to represent the present and historical states of the model as sets of weighted samples (particles) that are kept up-to-date as new readings arrive. We develop novel techniques to convert the queries on the model-based view directly into queries over particle tables, enabling highly efficient query processing. Finally, we present experimental evaluation of our prototype implementation over sensor data from the Intel Lab dataset that demonstrates the feasibility of online modeling of streaming data using our system and establishes the advantages of such tight integration between dynamic probabilistic models and database systems.

show abstract

Section: Dpms As Database Viewsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Online Filtering, Smoothing and Probabilistic Modeling of Streaming data

Kanagal

Deshpande

2008

2008 IEEE 24th International Conference on Data Engineering

116

View full text Add to dashboard Cite

show abstract

“…To make the retrieval of possible answers feasible, QPIAD bypasses the null value binding restriction by generating rewritten queries according to a set of mined attribute correlation rules. These rewritten queries are 1 Moreover, an attribute may not appear in a schema intentionally as the database manager may suppress the values of certain attributes. For example, the travel reservation website Priceline.com suppresses the airline/hotel name when booking tickets and hotel.…”

Section: Table 1: Statistics On Missing Values In Web Databasesmentioning

confidence: 99%

“…QPIAD processes a given selection query Q : σA m=vm according to the following two steps. 1. Send Q to the database and retrieve the base result set RS(Q) as the certain answers of Q.…”

Section: Query Rewriting Algorithmmentioning

confidence: 99%

“…Autos database thereby failing to return a possibly large portion of the relevant tuples. 1 User-defined Schemas: Another type of incompleteness occurs in the context of applications like Google Base [22] which allow users significant freedom to define and list their own attributes. This often leads to redundant attributes (e.g.…”

Section: Table 1: Statistics On Missing Values In Web Databasesmentioning

confidence: 99%

See 1 more Smart Citation

Query processing over incomplete autonomous databases: query rewriting using learned data dependencies

et al. 2009

View full text Add to dashboard Cite

Incompleteness due to missing attribute values (aka "null values") is very common in autonomous web databases, on which user accesses are usually supported through mediators. Traditional query processing techniques that focus on the strict soundness of answer tuples often ignore tuples with critical missing attributes, even if they wind up being relevant to a user query. Ideally we would like the mediator to retrieve such possible answers and gauge their relevance by accessing their likelihood of being pertinent answers to the query. The autonomous nature of web databases poses several challenges in realizing this objective. Such challenges include the restricted access privileges imposed on the data, the limited support for query patterns, and the bounded pool of database and network resources in the web environment. We introduce a novel query rewriting and optimization framework QPIAD that tackles these challenges. Our technique involves reformulating the user query based on mined correlations among the database attributes. The reformulated queries are aimed at retrieving the relevant possible answers in addition to the certain answers. QPIAD is able to gauge the relevance of such queries allowing tradeoffs in reducing the costs of database query processing and answer transmission. To support this framework, we develop methods for mining attribute correlations (in terms of Approximate Functional Dependencies), value distributions (in the form of Naïve Bayes Classifiers), and selectivity estimates. We present empirical studies to demonstrate that our approach is able to effectively retrieve relevant possible answers with high precision, high recall, and manageable cost.

show abstract