Online aggregation

Abstract

Aggregation in traditional datab=e systems is performed in batch mode: a query is submitted, the system processes a large volume of data over a long period of time, and, eventually, the final answer is returned. This archaic approach is frustrating to users and has been abandoned in most other areas of computing. In this paper we propose a new online aggregation interface that permits users to both observe the progress of their aggregation queries and control execution on the fly. After outlining usability and performance requirements for a system supporting online aggregation, we present a suite of techniques that extend a database system to meet these requirements, These include methods for returning the output in random order, for providing control over the relative rate at which different aggregates are computed, and for computing running confidence intervals. Finally, we report on an initial implementation of online aggregation in POSTGRES.

Key Constraints and Monotonic Aggregates in Deductive Databases

“…Online aggregation [8], in particular, cannot be expressed under the current approach that relies on a totally ordered universe to sort the elements of the set being processed, starting from its least element. In fact, at the core of on-line aggregation, there is the idea of returning partial results after visiting a proper subset of the given dataset, while the rest is still unknown.…”

“…There has been much recent interest in online aggregates [8], which also find important applications in logic programming, as discussed later in this paper. For instance, when computing averages on non-skewed data, the aggregate often converges toward the final value long before all the elements in the set are visited.…”

“…For instance, when computing averages on non-skewed data, the aggregate often converges toward the final value long before all the elements in the set are visited. Thus, the system should support early returns to allow the user to check convergence and stop the computation as soon as the series of successive values has converged within the prescribed accuracy [8]. UDAs with early returns can be defined in LDL++ through the use of ereturn rules.…”

“…In a more general framework, the user would want to control how often new results are to be returned to the user, on the basis of the estimated progress toward convergence in the computation [8]. UDAs provide a natural setting for this level of control.…”

Limiting Result Cardinalities for Multidatabase Queries Using Histograms

“…An interactive and iterative way to provide approximate answers for aggregated queries, called online-aggregation, is described in [HHW97]. Here the user starts with a relative imprecise answer provided by a first small random sample of the data.…”