Eddies

Abstract

In large federated and shared-nothing databases, resources can exhibit widely fluctuating characteristics. Assumptions made at the time a query is submitted will rarely hold throughout the duration of query processing. As a result, traditional static query optimization and execution techniques are ineffective in these environments. In this paper we introduce a query processing mechanism called an eddy , which continuously reorders operators in a query plan as it runs. We characterize the moments of symmetry during which pipelined joins can be easily reordered, and the synchronization barriers that require inputs from different sources to be coordinated. By combining eddies with appropriate join algorithms, we merge the optimization and execution phases of query processing, allowing each tuple to have a flexible ordering of the query operators. This flexibility is controlled by a combination of fluid dynamics and a simple learning algorithm. Our initial implementation demonstrates promising results, with eddies performing nearly as well as a static optimizer/executor in static scenarios, and providing dramatic improvements in dynamic execution environments.

Optimization and Execution of Complex Scientific Queries over Uncorrelated Experimental Data

“…The groups are then greedily reordered based on the measured estimates of the group costs and selectivities. To minimize overhead the processing plan is reoptimized once for every read complex stream object rather than for every query plan operator; there is no dynamic reordering per operator as with Eddies [3].…”

“…[1,2,3,6,13,17,19,20]) continuously adapt the execution plan of a query to reflect significant changes in data statistics. By contrast SQISLE profiles a stream until statistical properties of the streamed objects are stabilized, and then reoptimizes the query using the stable statistics.…”

“…Such monitoring for each simple data element adds significant overhead for large queries over complex objects. For Eddies [3] this overhead is even more significant, since optimization is performed whenever a tuple is scheduled for a next operator. To avoid the high cost of monitoring individual data elements, in SQISLE the profile-controller operator monitors the execution of an entire plan once per complex input stream object until sufficient statistics are collected about the objects, after which the plan is dynamically replaced with a final plan.…”

Query Processing in RDF/S-Based P2P Database Systems

“…Furthermore, we plan to study the tradeoff between result completeness and response time of queries using appropriate information quality metrics (e.g., coverage of schema classes and properties [12,33,32,18]) enabling to obtain quickly the Top-K answers [27,37]. Finally, we plan to consider adaptive implementations of algebraic operators borrowing ideas from [3,19,22].…”

Flux: an adaptive partitioning operator for continuous query systems

“…A key feature of these systems is to treat queries like data and index them to take advantage of their commonality. The latter two systems employ an Eddy [24] to dynamically adapt a query plan to changing workload characteristics.…”