Open issues in parallel query optimization

Hasan, Waqar; Florescu, Daniela; Valduriez, Patrick

doi:10.1145/234889.234894

Cited by 36 publications

(19 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Over the years, this has led to the development of a host of query processing and optimization algorithms for parallel databases, aimed toward maximizing the query-throughput of the system or minimizing the response time of a single large query. Broadly speaking, the parallelism in a parallel database can be exploited in three ways during query processing [15], [20]. Different query operators that do not depend on each other can be executed in parallel on different processors (independent parallelism).…”

Section: Introductionmentioning

confidence: 99%

Flow Algorithms for Parallel Query Optimization

Deshpande

Hellerstein²

2008

2008 IEEE 24th International Conference on Data Engineering

View full text Add to dashboard Cite

Abstract-We address the problem of minimizing the response time of a multi-way join query using pipelined (inter-operator) parallelism, in a parallel or a distributed environment. We observe that in order to fully exploit the parallelism in the system, we must consider a new class of "interleaving" plans, where multiple query plans are used simultaneously to minimize the response time of a query (or to maximize the tuple-throughput of the system). We cast the query planning problem in this environment as a "flow maximization problem", and present polynomial-time algorithms that (statically) find the optimal set of plans to use for a given query, for a large class of multiway join queries. Our proposed algorithms also naturally extend to query optimization over web services. Finally we present an extensive experimental evaluation that demonstrates both the need to consider such plans in parallel query processing and the effectiveness of our algorithms.

show abstract

Section: Introductionmentioning

confidence: 99%

Flow Algorithms for Parallel Query Optimization

Deshpande

Hellerstein²

2008

2008 IEEE 24th International Conference on Data Engineering

View full text Add to dashboard Cite

show abstract

“…Therefore, the DR-DPDP proof generation and update processes are distributed over some servers running in parallel, leading to better performance. This is similar to the shared-nothing architecture that leads to the parallel execution of commands, which in turn will reduce the response time and improve the scalability [22].…”

Section: Security Of Dr-dpdpmentioning

confidence: 99%

Transparent, Distributed, and Replicated Dynamic Provable Data Possession

Etemad

Küpçü

2013

Lecture Notes in Computer Science

View full text Add to dashboard Cite

With the growing trend toward using outsourced storage, the problem of efficiently checking and proving data integrity needs more consideration. Starting with PDP and POR schemes in 2007, many cryptography and security researchers have addressed the problem. After the first solutions for static data, dynamic versions were developed (e.g., DPDP). Researchers also considered distributed versions of such schemes. Alas, in all such distributed schemes, the client needs to be aware of the structure of the cloud, and possibly pre-process the file accordingly, even though the security guarantees in the real world are not improved.We propose a distributed and replicated DPDP which is transparent from the client's viewpoint. It allows for real scenarios where the cloud storage provider (CSP) may hide its internal structure from the client, flexibly manage its resources, while still providing provable service to the client. The CSP decides on how many and which servers will store the data. Since the load is distributed on multiple servers, we observe one-to-two orders of magnitude better performance in our tests, while availability and reliability are also improved via replication. In addition, we use persistent rank-based authenticated skip lists to create centralized and distributed variants of a dynamic version control system with optimal complexity.

show abstract

“…The challenges and open issues of query optimization in parallel database systems were highlighted by Hasan et al [11]. While considerable progress has been made over the past decade, parallel query optimization remains an active field of research fueled by the advent of data-intensive business intelligence and data warehousing applications.…”

Section: Related Workmentioning

confidence: 99%

“…One approach is to split the operation into two phases [11,12]: serial optimization followed by parallelization. The first phase hides the parallelism metrics from the optimizer and involves finding the optimal serial QEP as in a non-parallel environment.…”

Section: Related Workmentioning

confidence: 99%

Progressive optimization in a shared-nothing parallel database

Han

Markl

et al. 2007

Proceedings of the 2007 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

Commercial enterprise data warehouses are typically implemented on parallel databases due to the inherent scalability and performance limitation of a serial architecture. Queries used in such large data warehouses can contain complex predicates as well as multiple joins, and the resulting query execution plans generated by the optimizer may be suboptimal due to mis-estimates of row cardinalities. Progressive optimization (POP) is an approach to detect cardinality estimation errors by monitoring actual cardinalities at runtime and to recover by triggering re-optimization with the actual cardinalities measured. However, the original serial POP solution is based on a serial processing architecture, and the core ideas cannot be readily applied to a parallel shared-nothing environment. Extending the serial POP to a parallel environment is a challenging problem since we need to determine when and how we can trigger re-optimization based on cardinalities collected from multiple independent nodes. In this paper, we present a comprehensive and practical solution to this problem, including several novel voting schemes whether to trigger re-optimization, a mechanism to reuse local intermediate results across nodes as a partitioned materialized view, several flavors of parallel checkpoint operators, and parallel checkpoint processing methods using efficient communication protocols. This solution has been prototyped in a leading commercial parallel DBMS. We have performed extensive experiments using the TPC-H benchmark and a real-world database. Experimental results show that our solution has negligible runtime overhead and accelerates the performance of complex OLAP queries by up to a factor of 22.

show abstract

Open issues in parallel query optimization

Abstract: We provide an overview of query processing in parallel database systems and discuss several open issues in the optimization of queries for parallel machines.

Cited by 36 publications

References 18 publications

Flow Algorithms for Parallel Query Optimization

Flow Algorithms for Parallel Query Optimization

Transparent, Distributed, and Replicated Dynamic Provable Data Possession

Progressive optimization in a shared-nothing parallel database

Contact Info

Product

Resources

About