PIP: A database system for great and small expectations

Abstract

Estimation via sampling out of highly selective join queries is well known to be problematic, most notably in online aggregation. Without goal-directed sampling strategies, samples falling outside of the selection constraints lower estimation efficiency at best, and cause inaccurate estimates at worst. This problem appears in general probabilistic database systems, where query processing is tightly coupled with sampling. By committing to a set of samples before evaluating the query, the engine wastes effort on samples that will be discarded, query processing that may need to be repeated, or unnecessarily large numbers of samples. We describe PIP, a general probabilistic database system that uses symbolic representations of probabilistic data to defer computation of expectations, moments, and other statistical measures until the expression to be measured is fully known. This approach is sufficiently general to admit both continuous and discrete distributions. Moreover, deferring sampling enables a broad range of goal-oriented sampling-based (as well as exact) integration techniques for computing expectations, allows the selection of the integration strategy most appropriate to the expression being measured, and can reduce the amount of sampling work required. We demonstrate the effectiveness of this approach by showing that even straightforward algorithms can make use of the added information. These algorithms have a profoundly positive impact on the efficiency and accuracy of expectation computations, particularly in the case of highly selective join queries.

Join queries on uncertain data: Semantics and efficient processing

“…RELATED WORK There has been substantial work on uncertain data management lately (e.g., [10,28,24,16,17]). Cheng et al [7] proposed probabilistic threshold join, which is the same as our v-join semantics.…”

Uncertainty Annotated Databases - A Lightweight Approach for Approximating Certain Answers

“…Green et al [24] studied probabilistic versions of C-tables. Virtual C-tables generalize C-tables [30,49] by allowing symbolic expressions as values.…”

Dissociation and propagation for approximate lifted inference with standard relational database management systems

“…Query evaluation over probabilistic databases corresponds to solving the weighted model counting problem, and current approaches can be classified into three categories ( Fig. 20): (1) incomplete approaches identify tractable cases either at the query-level [13,14,24,54] or the data-level [53,65,69] and ignore the rest; (2) exact approaches [2,43,68] are based on variants and extensions of a complete search based on the DPLL procedure [35] and work well for queries over databases with simple lineage expressions, but perform poorly on complex lineage expressions; and (3) approximate approaches usually first compute the lineage of the query on the given database to obtain a Boolean formula, then either apply variants of Monte Carlo sampling methods [42,45,46,63], or approximate the number of models of the Boolean lineage expression [23,55,64]. A recent approach combines safe plans with Monte Carlo simulation [38].…”