Efficient and accurate cost models for parallel query optimization (extended abstract)

Ganguly, Šumit; Goel, Akshay; Silberschatz, Abraham

doi:10.1145/237661.237707

Cited by 11 publications

(13 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If, instead, the configuration is partially specified in line 10, the value of cost represents a lower bound on the cost of any configuration derived from the new one. If cost is worse than that of the incumbent, we prune the new configuration (lines [16][17], because no solution derived from it would be better than the current incumbent. When the stopping condition is met, we return in line 18 the current incumbent solution, which satisfies the storage constraint with minimum cost among the explored partition configurations.…”

Section: Table/column Selection Policymentioning

confidence: 99%

See 1 more Smart Citation

Automated partitioning design in parallel database systems

Nehme

Bruno

2011

Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

In recent years, Massively Parallel Processors (MPPs) have gained ground enabling vast amounts of data processing. In such environments, data is partitioned across multiple compute nodes, which results in dramatic performance improvements during parallel query execution. To evaluate certain relational operators in a query correctly, data sometimes needs to be re-partitioned (i.e., moved) across compute nodes. Since data movement operations are much more expensive than relational operations, it is crucial to design a suitable data partitioning strategy that minimizes the cost of such expensive data transfers. A good partitioning strategy strongly depends on how the parallel system would be used. In this paper we present a partitioning advisor that recommends the best partitioning design for an expected workload. Our tool recommends which tables should be replicated (i.e., copied into every compute node) and which ones should be distributed according to specific column(s) so that the cost of evaluating similar workloads is minimized. In contrast to previous work, our techniques are deeply integrated with the underlying parallel query optimizer, which results in more accurate recommendations in a shorter amount of time. Our experimental evaluation using a real MPP system, Microsoft SQL Server 2008 Parallel Data Warehouse, with both real and synthetic workloads shows the effectiveness of the proposed techniques and the importance of deep integration of the partitioning advisor with the underlying query optimizer.

show abstract

Section: Table/column Selection Policymentioning

confidence: 99%

“…The second broad area that relates to our work includes distributed architectures and distributed query processing [15,17,7,20,16,30,24,28,6]. While most of these techniques focus on optimizing parallel query execution in distributed environments, largely they assume that databases have been optimally partitioned to start with.…”

Section: Related Workmentioning

confidence: 99%

Automated partitioning design in parallel database systems

Nehme

Bruno

2011

Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

show abstract

“…At last, queries belong to the same set will have the same representative, which means the vertices they denote are connected directly to the root vertex. Finally, we create query groups for each disjoint set and add corresponding queries that belong to this set (lines [14][15][16][17][18][19][20][21]. These groups are the query sets that we can directly apply heuristic based MQO algorithms afterward.…”

Section: Grouping Methodsmentioning

confidence: 99%

Query grouping–based multi‐query optimization framework for interactive SQL query engines on Hadoop

Chen

Lin

Wang³

et al. 2018

Concurrency and Computation

View full text Add to dashboard Cite

In the past few years, executing high-concurrency queries with interactive SQL query engines on Hadoop has become an important activity for many organizations. However, these systems do not adopt Multi-Query Optimization (MQO) to accelerate the process. There are two major concerns.Firstly, traditional MQO researches assume that multiple queries have high similarity. However, these systems usually serve a variety of applications. Although queries from the same application have high similarity, queries from different applications may have low similarity, so using traditional MQO will be inefficient and time consuming. Secondly, integrating MQO may lead to lots of system modifications. To integrate MQO into interactive SQL query engines on Hadoop efficiently, a query grouping-based MQO framework is proposed. A lightweight mechanism is used to represent SQL queries, on which a grouping method is exploited to speed up the optimization process. A cost model is integrated to estimate the execution cost of interactive SQL query engines on Hadoop. By using the proposed framework, we modify Impala system to support MQO, and the experimental results on TPC-DS show significant performance improvements. KEYWORDSgrouping method, Impala system, multi-query optimization INTRODUCTIONLarge-scale analytical data processing has become a commonplace in many enterprises and research groups. Many interactive SQL query engines on Hadoop are designed to deal with the scenario. Query processing on large-scale analytical data is introduced by Google Dremel system. 1 Since then, several such systems appear, eg, Apache Impala system, 2 Apache Hawq system, 3 Apache Drill system, 4 and Facebook Presto system. These systems usually support a wide range of applications concurrently and have a large throughput.Multi-Query Optimization (MQO) 5 is a well-known database research problem, which has been studied by database community since the 1980s.The most important problem of MQO is to take full advantage of common tasks by constructing a global plan. In order to solve this problem, many algorithms have been proposed, including heuristic algorithms 6-8 and some improved algorithms. [9][10][11] There are two alternative system architectures that can be used for integrating MQO. One is to take full advantage of the local optimizer, which can only optimize a single query at a time and merge the optimal plans of all queries together. The other is to modify the optimizer to process a set of queries together and generate the optimal plan of all queries directly. However, an MQO module may need to be developed from scratch for this architecture. 5As far as we know, none of the interactive SQL query engines on Hadoop chooses to integrate these classical MQO algorithms for performance improvement. There are two main reasons. Firstly, traditional MQO algorithms assume that multiple queries have high similarity. However, these systems usually serve a variety of applications. Although queries from the same application have high similarity, queries from differ...

show abstract

“…In summary, compared to existing cost models for parallel query runtime estimation [13,12] and fault-tolerance in streaming engines [21], our models capture the dynamic operator interactions in pipelined queries, which we observed to affect runtime predictions and fault-tolerance optimization. For example, a fast operator following a slow one in a pipeline will produce its output slowly.…”

Section: Modeling Recovery Timementioning

confidence: 99%

A latency and fault-tolerance optimizer for online parallel query plans

Upadhyaya

Kwon

Balazinska

2011

Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

We address the problem of making online, parallel query plans fault-tolerant: i.e., provide intra-query fault-tolerance without blocking. We develop an approach that not only achieves this goal but does so through the use of different fault-tolerance techniques at different operators within a query plan. Enabling each operator to use a different faulttolerance strategy leads to a space of fault-tolerance plans amenable to cost-based optimization. We develop FTOpt, a cost-based fault-tolerance optimizer that automatically selects the best strategy for each operator in a query plan in a manner that minimizes the expected processing time with failures for the entire query. We implement our approach in a prototype parallel query-processing engine. Our experiments demonstrate that (1) there is no single best fault-tolerance strategy for all query plans, (2) often hybrid strategies that mix-and-match recovery techniques outperform any uniform strategy, and (3) our optimizer correctly identifies winning fault-tolerance configurations.

show abstract

Efficient and accurate cost models for parallel query optimization (extended abstract)

Cited by 11 publications

References 21 publications

Automated partitioning design in parallel database systems

Automated partitioning design in parallel database systems

Query grouping–based multi‐query optimization framework for interactive SQL query engines on Hadoop

A latency and fault-tolerance optimizer for online parallel query plans

Contact Info

Product

Resources

About