Eddies

Abstract

In large federated and shared-nothing databases, resources can exhibit widely fluctuating characteristics. Assumptions made at the time a query is submitted will rarely hold throughout the duration of query processing. As a result, traditional static query optimization and execution techniques are ineffective in these environments. In this paper we introduce a query processing mechanism called an eddy, which continuously reorders operators in a query plan as it runs. We characterize the moments of symmetry during which pipelined joins can be easily reordered, and the synchronization barriers that require inputs from different sources to be coordinated. By combining eddies with appropriate join algorithms, we merge the optimization and execution phases of query processing, allowing each tuple to have a flexible ordering of the query operators. This flexibility is controlled by a combination of fluid dynamics and a simple learning algorithm. Our initial implementation demonstrates promising results, with eddies performing nearly as well as a static optimizer/executor in static scenarios, and providing dramatic improvements in dynamic execution environments.There is increasing interest in query engines that run at unprecedented scale, both for widely-distributed information resources, and for massively parallel database systems. We are building a system called Telegraph, which is intended to run queries over all the data available on line. A key requirement of a large-scale system like Telegraph is that it function robustly in an unpredictable and constantly fluctuating environment. This unpredictability is endemic in large-scale systems, because of increased complexity in a number of dimensions: Hardware and Workload Complexity: In wide-area environments, variabilities are commonly observable in the bursty performance of servers and networks [UFA98]. These systems often serve large communities of users whose aggregate behavior can be hard to predict, and the hardware mix in the wide area is quite heterogeneous. Large clusters of computers can exhibit similar performance variations, due to a mix of user requests and heterogeneous hardware evolution. Even in totally homogeneous environments, hardware performance can be unpredictable: for example, the outer tracks of a disk can exhibit almost twice the bandwidth of inner tracks [Met97]. Data Complexity: Selectivity estimation for static alphanu-Figure 1: An eddy in a pipeline. Data flows into the eddy from input relations & ( ' ¦ ) and 0. The eddy routes tuples to operators; the operators run as independent threads, returning tuples to the eddy. The eddy sends a tuple to the output only when it has been handled by all the operators. The eddy adaptively chooses an order to route each tuple through the operators. meric data sets is fairly well understood, and there has been initial work on estimating statistical properties of static sets of data with complex types [Aok99] and methods [BO99]. But federated data often comes without any statistical summaries, and complex n...

Challenges in Ubiquitous Data Management

“…Telegraph uses a novel approach to query execution based on "eddies", which are dataflow control structures that route data to query operators on an item -by-item basis [AH00]. Telegraph does not rely upon a traditional query plan, but rather, allows the "plan" to develop and adapt during the execution.…”

Just-In-Time Processing of Continuous Queries

“…If no matching tuple of a 1 appears in C before the expiration of these intermediate results, the resources spent on them are wasted 1 . Next, suppose that at time 2 a new record b 4 , matching a 1 , arrives while there are still no join partners of a 1 in S C . By probing S A with b 4 , Op 1 generates a partial result a 1 b 4 , and subsequently performs a futile probing against S C .…”

“…Next, suppose that at time 2 a new record b 4 , matching a 1 , arrives while there are still no join partners of a 1 in S C . By probing S A with b 4 , Op 1 generates a partial result a 1 b 4 , and subsequently performs a futile probing against S C . Similar to the previous three partial results, the computation of a 1 b 4 is a waste of resources.…”

“…RELATED WORK Existing work in the data stream literature can be classified into two categories: the first aims at summarizing streaming data into synopsis structures (e.g., histograms, wavelets, sketches) and using them to provide fast, approximate answers to specific aggregate queries (e.g., [15,12]); the second focuses on the design of general-purpose DSMSs (e.g., Aurora [2,8], STREAM [1], TelegraphCQ [4], etc.) with formal semantics, expressive query languages and efficient query processing techniques.…”

“…For example, the query plan in Figure 1b [11] provides a tradeoff between memory and CPU resources, resembling a tree of MJoin operators. Finally, the Eddy architecture [4], shown in Figure 2b, features the novel Eddy operator that dynamically routes source tuples and intermediate results to appropriate operators to complete their processing. The proposed algorithms can be applied to all these types of join plans.…”