Sampling-based estimators for subset-based queries

Joshi, Shantanu; Jermaine, Christopher

doi:10.1007/s00778-008-0095-0

Cited by 17 publications

(11 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is because Verdict reaches a target error bound much earlier by combining its model with the raw answer of the AQP engine. [6,19,24,36,45,66,87]: Instead of continuously refining approximate answers and reporting them to the user, these engines simply take a time-bound from the user, and then they predict the largest sample size that they can process within the requested time-bound; thus, they minimize error bounds within the allotted time. For these engines, Verdict simply replaces the user's original time bound t 1 with a slightly smaller value t 1 − ǫ before passing it down to the AQP engine, where ǫ is the time needed by Verdict for inferring the improved answer and improved error.…”

Section: Deployment Scenariosmentioning

confidence: 99%

Database Learning

Park

Tajik

Cafarella

et al. 2017

Proceedings of the 2017 ACM International Conference on Management of Data

View full text Add to dashboard Cite

In today's databases, previous query answers rarely benefit answering future queries. For the first time, to the best of our knowledge, we change this paradigm in an approximate query processing (AQP) context. We make the following observation: the answer to each query reveals some degree of knowledge about the answer to another query because their answers stem from the same underlying distribution that has produced the entire dataset. Exploiting and refining this knowledge should allow us to answer queries more analytically, rather than by reading enormous amounts of raw data. Also, processing more queries should continuously enhance our knowledge of the underlying distribution, and hence lead to increasingly faster response times for future queries.We call this novel idea-learning from past query answersDatabase Learning. We exploit the principle of maximum entropy to produce answers, which are in expectation guaranteed to be more accurate than existing sample-based approximations. Empowered by this idea, we build a query engine on top of Spark SQL, called Verdict. We conduct extensive experiments on real-world query traces from a large customer of a major database vendor. Our results demonstrate that Verdict supports 73.7% of these queries, speeding them up by up to 23.0× for the same accuracy level compared to existing AQP systems.

show abstract

Section: Deployment Scenariosmentioning

confidence: 99%

Database Learning

Park

Tajik

Cafarella

et al. 2017

Proceedings of the 2017 ACM International Conference on Management of Data

View full text Add to dashboard Cite

show abstract

“…For example, Charikar et al [15] study distinct value estimation from a sample; Joshi and Jermaine [25] propose an EM algorithm to quantify aggregate queries with subset testing.…”

Section: Related Workmentioning

confidence: 99%

“…There has been a large body of research on using sampling to provide quick answers to database queries, on database systems [9,15,16,22,23,24,25,33,44], and data stream systems [12,31]. Approximate aggregate processing has been the focus of many of these works, which study randomized joins [24], optimal sample construction [9,16], sample reusing [44], and sampling plan in a stream setting [12,31].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

The analytical bootstrap

Zeng

Shi

Mozafari

et al. 2014

Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

Sampling is one of the most commonly used techniques in Approximate Query Processing (AQP)-an area of research that is now made more critical by the need for timely and cost-effective analytics over "Big Data". Assessing the quality (i.e., estimating the error) of approximate answers is essential for meaningful AQP, and the two main approaches used in the past to address this problem are based on either (i) analytic error quantification or (ii) the bootstrap method. The first approach is extremely efficient but lacks generality, whereas the second is quite general but suffers from its high computational overhead. In this paper, we introduce a probabilistic relational model for the bootstrap process, along with rigorous semantics and a unified error model, which bridges the gap between these two traditional approaches. Based on our probabilistic framework, we develop efficient algorithms to predict the distribution of the approximation results. These enable the computation of any bootstrap-based quality measure for a large class of SQL queries via a single-round evaluation of a slightly modified query. Extensive experiments on both synthetic and real-world datasets show that our method has superior prediction accuracy for bootstrap-based quality measures, and is several orders of magnitude faster than bootstrap.

show abstract

“…There are many other valuable works [29][30][31][32] on query optimization and sampling, but none of them deals with the problem of window functions. In this paper, we study how to evaluate online aggregate for queries involving and extend the existing work [1] for new application scenarios.…”

Section: Related Workmentioning

confidence: 99%

Approximate Calculation of Window Aggregate Functions via Global Random Sample

Song

Liu

et al. 2018

Data Sci. Eng.

View full text Add to dashboard Cite

Window functions have been a part of the SQL standard since 2003 and have been studied extensively during the past decade. They are widely used in data analysis; almost all the current mainstream commercial databases support window functions. However, in recent years the size of datasets is growing steeply; the existing window function implementations are not efficient enough. Recently, some sampling-based algorithms (e.g., online aggregation) are proposed to deal with large and complex data in relational databases, which offer us a flexible trade-off between accuracy and efficiency. However, few sampling techniques has been considered for window functions in databases. In this paper, we extend our previous work (Song et al. in Asia-Pacific web and web-age information management joint conference on web and big data, Springer, pp 229-244, 2017) and proposed two new algorithms: range-based global sampling algorithm and rowlabeled sampling algorithm. The proposed algorithms use global sampling rather than local sampling and are more efficient than other existing algorithms. And we find our proposed algorithms out performed the baseline method over the TPC-H benchmark dataset.

show abstract

Sampling-based estimators for subset-based queries

Cited by 17 publications

References 22 publications

Database Learning

Database Learning

The analytical bootstrap

Approximate Calculation of Window Aggregate Functions via Global Random Sample

Contact Info

Product

Resources

About