Scheduling to minimize staleness and stretch in real-time data warehouses

Abstract

We study scheduling algorithms for loading data feeds into real time data warehouses, which are used in applications such as IP network monitoring, online financial trading, and credit card fraud detection. In these applications, the warehouse collects a large number of streaming data feeds that are generated by external sources and arrive asynchronously. Data for each table are generated at a constant rate, different tables possibly at different rates. For each data feed, the arrival of new data triggers an update that appends the new data to the corresponding table; if multiple updates are pending for the same table, they are batched together before being loaded. At time τ , if a table has been updated with information up to time r ≤ τ , its staleness is defined as τ − r.Our first objective is to schedule the updates on one or more processors in a way that minimizes the total staleness. In order to ensure fairness, our second objective is to limit the maximum "stretch", which we define (roughly) as the ratio between the duration of time an update waits till it is finished being processed, and the length of the update.In contrast to earlier work proving the nonexistence of constant-competitive algorithms for related scheduling problems, we prove that any online nonpreemptive algorithm, no processor of which is ever voluntarily idle, incurs a staleness at most a constant factor larger than an obvious lower bound on total staleness (provided that the processors are sufficiently fast). We give a constant-stretch algorithm, provided that the processors are sufficiently fast, for the quasiperiodic model, in which tables can be clustered into a few groups such that the update frequencies within each group * Work done while the author was visiting AT&T LabsResearch.Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SPAA '09, August 11-13, 2009, Calgary, Alberta, Canada. Copyright 2009 vary by at most a constant factor. Finally, we show that our constant-stretch algorithm is also constant-competitive (subject to the same proviso on processor speed) in the quasiperiodic model with respect to total weighted staleness, where tables are assigned weights that reflect their priorities.

UpStream: storage-centric load management for streaming applications with update semantics

“…If the EB is not empty at this time, we say that the window that is already in the EB is overwritten. This is what happens with windows [3,7] and [5,9] in our example. At the scheduling points, the DB will pop the interval in the EB (if any), which represents the most recently arrived full window.…”

“…The top of the WINS list contains the oldest started window (e.g., [3,5)). When this window closes (e.g., [3,7]), it is popped out of the WINS list and placed in EB. If the EB is not empty at this time, we say that the window that is already in the EB is overwritten.…”

“…At the scheduling points, the DB will pop the interval in the EB (if any), which represents the most recently arrived full window. For instance, [1,5] and [7,11] are the windows that the buffer manager commits to.…”

“…Window updates happen on window starters. The EB is always emptied when a new window starter arrives (e.g., 3,5,7,9,11). At the scheduling points, the DB will pop the interval in the EB (if any), which represents the most recently arrived partial window.…”

“…The staleness problem has been approached previously in web databases, soft real-time databases, or streaming data warehouses [2,7,10,15,16]. Basically, that research considered scheduling updates to base data or synchronizing local copies of remote databases while analyzing the tradeoffs between data timeliness (i.e., staleness) and response time of user queries.…”