A Lightweight Online Framework For Query Progress Indicators

Mishra, Chaitanya; Koudas, Nick

doi:10.1109/icde.2007.368996

Cited by 15 publications

(18 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous work in this area [8] [9][10] [11] [12] focused mostly on single site execution and heavily relied on previously collected operator statistics, which aren't always available in the largely User-Function based MapReduce paradigm. Only very recently [13] [14] has work appeared that addressed execution in MapReduce frameworks and even this work relied on pre-computation debug runs and was not directed at the streaming, dynamically provisioned, setting we employ.…”

Section: E Dealing With Perturbationsmentioning

confidence: 99%

“…Query progress estimation is a widely studied subject for single site query execution [8] [9][10] [11] [12], however only very recently did alternatives appear for large-scale parallel computations. Parallax [13] and its evolution, ParaTimer [14], represent the current state of the art in progress estimation for MapReduce pipelines.…”

Section: B Estimating Progress and Computing Resourcesmentioning

confidence: 99%

See 1 more Smart Citation

Deadline Queries: Leveraging the Cloud to Produce On-Time Results

Alves

Bizarro

Marques

2011

2011 IEEE 4th International Conference on Cloud Computing

View full text Add to dashboard Cite

MapReduce has become a widely used tool for computing complex tasks that process massive amounts of data in large clusters. Support for MapReduce tasks in cloud environments has been provided but it is left to users to make best guesses on the number of nodes needed for a task to complete within acceptable time. Moreover, the time a task will take to complete is often unknown beforehand. Previous research addressed this problem by establishing time constraints for query execution and, when needed, reduce the accuracy of queries using result approximation and/or sampling. However, in many situations reduced accuracy is not tolerable. In this paper we present FloodDQ, a MapReduce system that implements deadline queries-queries that must finish before a deadline, never discarding data or reducing accuracy. FloodDQ produces timely, accurate results by adaptively increasing or decreasing computing power, at runtime, towards completing execution within the specified deadline. In FloodDQ, users only specify a deadline and the input data. The system monitors the progress of the task and extrapolates whether it will complete on time. If the task is deemed to complete after the specified time, the system requests more nodes from an IaaS Cloud provider, and adds them to the computation. On the other hand, if the task is deemed to complete before the specified time the system quiesces and releases surplus nodes, cutting costs to a minimum. This paper describes FloodDQ's architecture for supporting deadline queries and presents experimental results where the system always meets the deadline in spite of changes to the number of nodes, size of data or existence of perturbations.

show abstract

Section: E Dealing With Perturbationsmentioning

confidence: 99%

Section: B Estimating Progress and Computing Resourcesmentioning

confidence: 99%

Deadline Queries: Leveraging the Cloud to Produce On-Time Results

Alves

Bizarro

Marques

2011

2011 IEEE 4th International Conference on Cloud Computing

View full text Add to dashboard Cite

show abstract

“…The first category includes work on progress estimation for running queries [5,15,16,18,19,22]. The key idea for this work is to collect runtime statistics from the actual execution of a query to dynamically predict the remaining work/time for the query.…”

Section: Related Workmentioning

confidence: 99%

Resource bricolage for parallel database systems

Naughton

Nehme

2014

Proc. VLDB Endow.

View full text Add to dashboard Cite

Running parallel database systems in an environment with heterogeneous resources has become increasingly common, due to cluster evolution and increasing interest in moving applications into public clouds. For database systems running in a heterogeneous cluster, the default uniform data partitioning strategy may overload some of the slow machines while at the same time it may under-utilize the more powerful machines. Since the processing time of a parallel query is determined by the slowest machine, such an allocation strategy may result in a significant query performance degradation.We take a first step to address this problem by introducing a technique we call resource bricolage that improves database performance in heterogeneous environments. Our approach quantifies the performance differences among machines with various resources as they process workloads with diverse resource requirements. We formalize the problem of minimizing workload execution time and view it as an optimization problem, and then we employ linear programming to obtain a recommended data partitioning scheme. We verify the effectiveness of our technique with an extensive experimental study on a commercial database system.

show abstract

“…The two follow-up papers [8], [16] on the WiscPI aim to increase the coverage of the progress indicator to a wider set of SQL queries and extend the single-query progress estimation to enable progress estimation for multiple queries. In [9] and [10], the authors propose a lightweight progress indicator, and they focus on improving cardinality estimation accuracy for various operators in the query plan. Since refining cardinality estimates is not the focus of the paper, we do not incorporate them into our progress indicator.…”

Section: Related Workmentioning

confidence: 99%

“…research institutes independently in the same conference. The variants [7], [8], [9], [10] of the MSRPI and the WiscPI, were proposed later to explore issues such as broadening the class of queries handled, or investigating the interaction between concurrent queries, or reducing cardinality estimation errors. Despite the passage of time, no work has addressed the quality of the original progress indicators on the problems for which they were proposed.…”

Section: Introductionmentioning

confidence: 99%

GSLPI: A Cost-Based Query Progress Indicator

Nehme

Naughton

2012

2012 IEEE 28th International Conference on Data Engineering

View full text Add to dashboard Cite

Abstract-Progress indicators for SQL queries were first published in 2004 with the simultaneous and independent proposals from Chaudhuri et al. and Luo et al. In this paper, we implement both progress indicators in the same commercial RDBMS to investigate their performance. We summarize common cases in which they are both accurate and cases in which they fail to provide reliable estimates. Although there are differences in their performance, much more striking is the similarity in the errors they make due to a common simplifying uniform future speed assumption. While the developers of these progress indicators were aware that this assumption could cause errors, they neither explored how large the errors might be nor did they investigate the feasibility of removing the assumption. To rectify this we propose a new query progress indicator, similar to these early progress indicators but without the uniform speed assumption. Experiments show that on the TPC-H benchmark, on queries for which the original progress indicators have errors up to 30X the query running time, the new progress indicator is accurate to within 10 percent. We also discuss the sources of the errors that still remain and shed some light on what would need to be done to eliminate them.

show abstract

A Lightweight Online Framework For Query Progress Indicators

Cited by 15 publications

References 11 publications

Deadline Queries: Leveraging the Cloud to Produce On-Time Results

Deadline Queries: Leveraging the Cloud to Produce On-Time Results

Resource bricolage for parallel database systems

GSLPI: A Cost-Based Query Progress Indicator

Contact Info

Product

Resources

About