Query processing over incomplete autonomous databases: query rewriting using learned data dependencies

Abstract

Incompleteness due to missing attribute values (aka "null values") is very common in autonomous web databases, on which user accesses are usually supported through mediators. Traditional query processing techniques that focus on the strict soundness of answer tuples often ignore tuples with critical missing attributes, even if they wind up being relevant to a user query. Ideally we would like the mediator to retrieve such possible answers and gauge their relevance by accessing their likelihood of being pertinent answers to the query. The autonomous nature of web databases poses several challenges in realizing this objective. Such challenges include the restricted access privileges imposed on the data, the limited support for query patterns, and the bounded pool of database and network resources in the web environment. We introduce a novel query rewriting and optimization framework QPIAD that tackles these challenges. Our technique involves reformulating the user query based on mined correlations among the database attributes. The reformulated queries are aimed at retrieving the relevant possible answers in addition to the certain answers. QPIAD is able to gauge the relevance of such queries allowing tradeoffs in reducing the costs of database query processing and answer transmission. To support this framework, we develop methods for mining attribute correlations (in terms of Approximate Functional Dependencies), value distributions (in the form of Naïve Bayes Classifiers), and selectivity estimates. We present empirical studies to demonstrate that our approach is able to effectively retrieve relevant possible answers with high precision, high recall, and manageable cost.

Data Completeness Measures

“…For example, Fox et al [24] addressed missing values from the entity attributes view, while Wolf et al [28,29] addressed missing values from the view of a tuple. As more than one view is available, we know that completeness is a multi-dimensional concept that requires some way to measure the missing units from the data set concerned.…”

“…We can also see from the definitions that the 'view(s)' of 'what is missing' was included by some researchers (notably [22][23][24][25][26][27][28][29]). For example, Fox et al [24] addressed missing values from the entity attributes view, while Wolf et al [28,29] addressed missing values from the view of a tuple.…”

“…To learn how completeness is measured, we surveyed the literature where completeness measures were proposed. Fox et al [24] 'The degree to which a data collection has values or all attributes of entities that are supposed to have values' Pipino et al [27] Column completeness: 'a function of the missing values in a column of a table' Wolf et al [28,29] 'Let R(A 1 , …, A n ) be a database relation. A tuple t ∊ R is said to be complete if it has non-null values for each of the attributes A i ; otherwise, it is considered incomplete.…”

BayesWipe: A multimodal system for data cleaning and consistent query answering on structured bigdata

“…The query rewriting part of this paper is inspired by the QPIAD system [23], but significantly improves upon it. QPIAD performed query rewriting over incomplete databases using approximate functional dependencies (AFD), and only cleaned data with null values, not wrong values.…”

Assessing relevance and trust of the deep web sources and results based on inter-source agreement

“…A number of methods have been tried for record linkage [Koudas et al 2006;Fellegi and Sunter 1969]. Completion and expansion of autonomous web database records at query time has been attempted [Gummadi et al 2011;Wolf et al 2009].…”