On random sampling over joins

Abstract

A major bottleneck in implementing sampling as a primitive relational operation is the inefficiency of sampling the output of a query. It is not even known whether it is possible to generate a sample of a join tree without first evaluating the join tree completely. We undertake a detailed study of this problem and attempt to analyze it in a variety of settings. We present theoretical results explaining the difficulty of this problem and setting limits on the efficiency that can be achieved. Based on new insights into the interaction between join and sampling, we develop join sampling techniques for the settings where our negative results do not apply. Our new sampling algorithms are significantly more efficient than those known earlier. We present experimental evaluation of our techniques on Microsoft's SQL Server 7.0.

Intelligent Network Client Profiler

“…The other method uses the packets arrival times (which can be used to anticipate a traffic burst) together with knowledge about the processing time required to process a sample. [23] proposes to use the least squares estimate and a certain set of heuristic rules to determine the sampling rate. The authors in [24] describe a flow sampling approach, which allows controlling the expected volume of samples and minimizes the variance of the estimates.…”

A Framework for the Physical Design Problem for Data Synopses

“…For join queries that access attributes from multiple datasets R 1 , ..., R l it is conceivable to construct a result approximation or result size estimation from multiple synopses. On the other hand, it is known that this approach may lead to unbounded approximation errors [5]. Therefore, we have adopted the approach of [1] to use special join synopses for this purpose.…”

“…As pointed out in [5] (in the context of sampling), it is usually not feasible to estimate arbitrary join queries from approximations of the joining base relations with acceptable accuracy. For sampling, this phenomenon is discussed extensively in [5], but it does also hold for all other data reduction techniques that estimate join queries from approximations of the base relations.…”

“…For sampling, this phenomenon is discussed extensively in [5], but it does also hold for all other data reduction techniques that estimate join queries from approximations of the base relations.…”

“…This broad importance of statistics management has led to a plethora of approximation techniques, for which [11] have coined the general term "data synopses": advanced forms of histograms [24,12,16], spline synopses [18,19], sampling [5,13,10], and parametric curve-fitting techniques [27,7] all the way to highly sophisticated methods based on kernel estimators [2] or Wavelets and other transforms [22,21,3]. However, most of these techniques take the local viewpoint of optimizing the approximation error for a single data distribution such as one database table with preselected relevant attributes.…”