Lower bounds for randomized read/write stream algorithms

Abstract

Motivated by the capabilities of modern storage architectures, we consider the following generalization of the data stream model where the algorithm has sequential access to multiple streams. Unlike the data stream model, where the stream is read only, in this new model (introduced in [8,9]) the algorithms can also write onto streams. There is no limit on the size of the streams but the number of passes made on the streams is restricted. On the other hand, the amount of internal memory used by the algorithm is scarce, similar to data stream model.We resolve the main open problem in [7] of proving lower bounds in this model for algorithms that are allowed to have 2-sided error. Previously, such lower bounds were shown only for deterministic and 1-sided error randomized algorithms [9,7]. We consider the classical set disjointness problem that has proved to be invaluable for deriving lower bounds for many other problems involving data streams and other randomized models of computation. For this problem, we show a near-linear lower bound on the size of the internal memory used by a randomized algorithm with 2-sided error that is allowed to have o(log N/ log log N ) passes over the streams. This bound is almost optimal since there is a simple algorithm that can solve this problem using logarithmic memory if the number of passes over the streams is allowed to be O(log N ).Applications include near-linear lower bounds on the internal memory for well-known problems in the literature: (1) approximately counting the number of distinct elements in the input (F0); (2) approximating the frequency of the mode of an input sequence (F * ∞ ); (3) computing the join of two relations; and (4) deciding if some node of an XML document matches an XQuery (or XPath) query. * Research supported by NSF grant CCF-0514870. † Part of this research was done while the author was visiting University of Washington ‡ Research supported in part by NSF grant CCF-0343672. Part of this research was done while the author was visiting IBM Almaden.Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Our techniques involve a novel direct-sum type of argument that yields lower bounds for many other problems. Our results asymptotically improve previously known bounds for any problem even in deterministic and 1-sided error models of computation.

Machine models for query processing

“…The above results easily lead to the following statements on the data complexity of relational algebra queries and queries posed against XML data [20,7] (for any choice of t 1 and ε with 0 < ε < 1).…”

“…It sometimes is convenient to adopt Beame, Jayram, and Rudra's [7] informal view of this as being a computation model where, in addition to some internal memory, a constant number of read/write streams (r/w, for short) are available; each such r/w stream corresponds to an external memory tape of the Turing machine. An illustration of the model is given in Figure 2.…”

“…For the set disjointness problem, in the presence of considerably less than log N head reversals, not even a randomized algorithm with 2-sided bounded error exists: Theorem 3.6 ( [7]) Let t be a positive interger and let ε be an arbitrary number with 0 < ε < 1. There is no (o(log N/ log log N ), O(N 1−ε ), t)-bounded randomized algorithm for the set disjointness problem on r/w streams such that • each "no"-instance is rejected with probability 2/3, • each "yes"-instance is accepted with probability 2/3.…”

Adaptive Matrix Vector Product

“…Our model shares some similarities with work in data streams with read/write tapes [5,6,10,11,12], though a key difference is that in our model we can only write the next symbol to the output tape and cannot read what we have already written. This is motivated in scenarios in which one wants to quickly transmit the part of the output that one has already computed, and one is too resource-constrained to keep this information locally.…”

Trading off space for passes in graph streaming problems

“…Lower bounds for sorting the input data and for other decision problems in this model have been proved in [Grohe et al 2006;. Very recently, these results have been extended to 2-sided error randomized algorithms [Beame et al 2007]. …”