Towards Optimal Moment Estimation in Streaming and Distributed Models

Abstract

One of the oldest problems in the data stream model is to approximate the p -th moment \(\Vert \mathbf {X}\Vert _p^p = \sum _{i=1}^n \mathbf {X}_i^p \) of an underlying non-negative vector \(\mathbf {X} \in \mathbb {R}^n \) , which is presented as a sequence of poly( n ) updates to its coordinates. Of particular interest is when p ∈ (0, 2]. Although a tight space bound of Θ (ϵ − 2 log n ) bits is known for this problem when both positive and negative updates are allowed, surprisingly there is still a gap in the space complexity of this problem when all updates are positive. Specifically, the upper bound is O (ϵ − 2 log n ) bits, while the lower bound is only Ω (ϵ − 2 + log n ) bits. Recently, an upper bound of \(\tilde{O}(\epsilon ^{-2} + \log n) \) bits was obtained under the assumption that the updates arrive in a random order . We show that for p ∈ (0, 1], the random order assumption is not needed. Namely, we give an upper bound for worst-case streams of \(\tilde{O}(\epsilon ^{-2} + \log n) \) bits for estimating \(\Vert \mathbf {X}\Vert _p^p \) . Our techniques also give new upper bounds for estimating the empirical entropy in a stream. On the other hand, we show that for p ∈ (1, 2], in the natural coordinator and blackboard distributed communication topologies, there is an \(\tilde{O}(\epsilon ^{-2}) \) bit max-communication upper bound based on a randomized rounding scheme. Our protocols also give rise to protocols for heavy hitters and approximate matrix product. We generalize our results to arbitrary communication topologies G , obtaining an \(\tilde{O}(\epsilon ^{2} \log d) \) max-communication upper bound, where d is the diameter of G . Interestingly, our upper bound rules out natural communication complexity-based approaches for proving an Ω (ϵ − 2 log n ) bit lower bound for p ∈ (1, 2] for streaming algorithms. In particular, any such lower bound must come from a topology with large diameter.